Filtering imaging system and applications thereof

ABSTRACT

The disclosure describes the use of image sensors in a variety of different imaging applications. In some implementations, the pixels of an image sensor may be configured (e.g., programmed) with one or more analog filters to sense a variety of things, including, for example: average intensity and color (like a standard camera), 3D depth (like a time of flight or structured light camera), changes and/or motion in an image (like an event based camera), spectral reflectance (like a spectroscopy camera), and many other features of an imaged scene or object. In alternative implementations, digital filtering may be applied to the output of an image sensor to realize one of these applications.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of and claims priority toU.S. patent application Ser. No. 15/678,032 filed Aug. 15, 2017, andtitled “FILTERING PIXELS AND USES THEREOF,” which claims priority toU.S. Provisional Patent Application No. 62/380,212 filed Aug. 26, 2016.

DESCRIPTION OF THE RELATED ART

Image sensors comprise a plurality of pixels that convert captured lightwithin an image into a current. Traditional pixels comprise aphotodetector and, with respect to active pixel sensors, an amplifier.The total light intensity falling on each photodetector is captured,without any differentiation between the wavelength or source of thelight. External filters, like Bayer filters, may be added to an imagesensor to filter out undesired light prior to reaching the photodetectorof each pixel.

BRIEF SUMMARY OF EMBODIMENTS

The technology described herein relates to a filtering imaging systemand applications thereof.

In a first example, an imaging system includes: a light source to outputan optical signal modulated with a code; an image sensor including aplurality of pixels; and a filter circuit to filter out signalsgenerated by one or more of the plurality of pixels from light that isnot modulated with the code and to pass signals generated by one or moreof the plurality of pixels from light that is modulated with the code.In some implementations, the system further includes: a light sourcedriver configured to output a modulated electrical signal to drive thelight source based on the modulation code.

In some implementations of the first example, each of the plurality ofpixels include: a photodiode; the filter circuit; and a read outfield-effect transistor (FET). In particular implementations, theplurality of pixels include: a first plurality of pixels, each of thefirst plurality of pixels to output a signal proportional to anintensity of light incident on a photodiode of the pixel; and a secondplurality of pixels, each of the second plurality of pixels to output asignal proportional to a change in intensity of light incident on aphotodiode of the pixel. In particular implementations, the plurality ofpixels include: a first plurality of pixels, each of the first pluralityof pixels to output a signal proportional to an intensity of lightincident on a photodiode of the pixel; a second plurality of pixels,each of the second plurality of pixels to output a signal proportionalto an increase in intensity of light incident on a photodiode of thepixel; and a third plurality of pixels, each of the third plurality ofpixels to output a signal proportional to a decrease in intensity oflight incident on a photodiode of the pixel.

In some implementations, the light source is to output a plurality ofoptical signals, each of the plurality of optical signals correspondingto a respective waveband or color of light modulated with a respectivecode, and each of the plurality of the pixels is configurable to detectone of the wavebands or colors of light. In particular implementations,each of the wavebands of light is phase modulated, and the filtercircuit of each of the plurality of the pixels is configured to filterthe wavebands of light based on their phase.

In some implementations of the first example, the filter circuit is adigital filter circuit to filter one or more digital signals output bythe image sensor. In particular implementations, the imaging systemfurther includes: a first filter circuit to output a signal proportionalto an increase in intensity of light incident on one or more of theplurality of pixels; and a second filter circuit to output a signalproportional to a decrease in intensity of light on one or more of theplurality of pixels.

In particular implementations, the imaging system includes: a firstlight source to output a first color of light modulated with a firstcode; a second light source to output a second color of light modulatedwith a second code; a first digital filter circuit to filter out signalsgenerated from light that is not modulated with the first code; and asecond digital filter circuit to filter out signals generated from lightthat is not modulated with the second code; and a digital filter circuitto combine outputs of the first digital filter circuit and seconddigital filter circuit to generate a composite color image.

In some implementations, the imaging system is a structured lightimaging system, and the imaging system further includes: a grating toshape the output optical signal modulated with a code.

In some implementations, the imaging system includes: a first lightsource to output a first optical signal modulated with a first code; asecond light source to output a second optical signal modulated with asecond code; a first filter circuit to filter out signals generated fromlight that is not modulated with the first code; and a second filtercircuit to filter out signals generated from light that is not modulatedwith the second code.

In some implementations, the imaging system is a laser scan imagingsystem, the light source includes a laser line scanner, and the imagingsystem includes: a first filter circuit to output a peak signal when aphase of light incident on one or more of the plurality of pixels of theimage sensor is a first value; a second filter circuit to output a peaksignal when a phase of light incident on one or more of the plurality ofpixels of the image sensor is a second value; a third filter circuit tooutput a peak signal when a phase of light incident on one or more ofthe plurality of pixels of the image sensor is a third value; and afourth filter circuit to output a peak signal when a phase of lightincident on one or more of the plurality of pixels of the image sensoris a fourth value. In some implementations, a frequency of each of theplurality of pixels is offset from a frequency of the optical signal.

In a second example, a method includes: generating a modulated opticalsignal at a light source of an imaging system, the modulated opticalsignal carrying a modulation code; configuring a filter circuit tofilter out signals generated by one or more pixels of a plurality ofpixels of an image sensor from light that is not modulated with the codeand to pass signals generated by one or more pixels of the image sensorfrom light that is modulated with the code; receiving light at the imagesensor; and using at least the configured filter circuit to filter thereceived light based on the modulation code.

In some implementations, the method further includes: configuring afirst set of the plurality of pixels to output a signal proportional toan intensity of light incident on a photodiode of the pixel; andconfiguring a second set of the plurality of pixels to output a signalproportional to a change in intensity of light incident on a photodiodeof the pixel.

In some implementations, the method further includes: configuring afirst filter circuit to output a signal proportional to an increase inintensity of light incident on one or more of the plurality of pixels;and configuring a second filter circuit to output a signal proportionalto a decrease in intensity of light on one or more of the plurality ofpixels.

In some implementations, the method further includes: configuring afirst set of the plurality of pixels to output a signal proportional toan intensity of light incident on a photodiode of the pixel; configuringa second set of the plurality of pixels to output a signal proportionalto an increase in intensity of light incident on a photodiode of thepixel; and configuring a third set of the plurality of pixels to outputa signal proportional to a decrease in intensity of light incident on aphotodiode of the pixel.

In some implementations, the method includes: generating a firstmodulated optical signal at a first light source of the imaging system,the first modulated optical signal carrying a first modulation code;generating a second modulated optical signal at a second light source ofthe imaging system, the second modulated optical signal carrying asecond modulation code; configuring a first filter circuit to filter outsignals generated from light that is not modulated with the first code;and configuring a second filter circuit to filter out signals generatedfrom light that is not modulated with the second code.

In some implementations, the imaging system is a structured lightimaging system, and the method further includes: shaping the modulatedoptical signal by passing it through a grating of the imaging system.

In some implementations, the method includes: generating a firstmodulated optical signal at the light source, the first modulatedoptical signal having a first waveband and carrying a first modulationcode; generating a second modulated optical signal at the first light,the second modulated optical signal having a second waveband andcarrying a second modulation code; and configuring the filter circuit ofeach of the plurality of pixels to detect light carrying the firstmodulation code or light carrying the second modulation code.

In some implementations, the imaging system is a laser scan imagingsystem, the light source includes a laser line scanner, and the methodfurther includes: configuring a first filter circuit to output a peaksignal when a phase of light incident on one or more of the plurality ofpixels of the image sensor is a first value; configuring a second filtercircuit to output a peak signal when a phase of light incident on one ormore of the plurality of pixels of the image sensor is a second value;configuring a third filter circuit to output a peak signal when a phaseof light incident on one or more of the plurality of pixels of the imagesensor is a third value; and configuring a fourth filter circuit tooutput a peak signal when a phase of light incident on one or more ofthe plurality of pixels of the image sensor is a fourth value.

In a third example, an imaging system includes: a light source to outputan optical signal; an image sensor comprising a plurality of pixels; andone or more filter circuits to add or subtract consecutively sampledsignals to output a composite signal including positive and negativeschanges in an intensity of light incident on one or more of theplurality of pixels. In some implementations, the consecutive sampledsignals include a plurality of consecutively captured image frames, theone or more filter circuits include one or more digital filter circuitsto add or substract the plurality of consecutively captured image framesto create a composite image frame, and the composite image framedisplays positive and negative changes in the intensity of lightincident on one or more of the plurality of the pixels during capture ofthe plurality of consecutively captured image frames. In otherimplementations of the third example, each of the plurality of pixelsinclude: a photodiode; the one or more filter circuits; and a read outfield-effect transistor (FET).

Other features and aspects of the disclosed method will become apparentfrom the following detailed description, taken in conjunction with theaccompanying drawings, which illustrate, by way of example, the featuresin accordance with embodiments of the disclosure. The summary is notintended to limit the scope of the claimed disclosure, which is definedsolely by the claims attached hereto.

It should be appreciated that all combinations of the foregoing concepts(provided such concepts are not mutually inconsistent) are contemplatedas being part of the inventive subject matter disclosed herein. Inparticular, all combinations of claimed subject matter appearing at theend of this disclosure are contemplated as being part of the inventivesubject matter disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The technology disclosed herein, in accordance with one or more variousembodiments, is described in detail with reference to the followingfigures. The drawings are provided for purposes of illustration only andmerely depict typical or example embodiments of the disclosedtechnology. These drawings are provided to facilitate the reader'sunderstanding of the disclosed technology and shall not be consideredlimiting of the breadth, scope, or applicability thereof. It should benoted that for clarity and ease of illustration these drawings are notnecessarily made to scale.

FIG. 1 is a circuit diagram of an example derivative pixel in accordancewith embodiments of the technology disclosed herein.

FIG. 2 illustrates another example derivative pixel for accumulatingcharge in accordance with embodiments of the technology disclosedherein.

FIG. 3 is a block diagram illustration of an example derivative pixel inaccordance with embodiments of the technology disclosed herein.

FIG. 4 is a block diagram illustration of another example derivativepixel in accordance with embodiments of the technology disclosed herein.

FIGS. 5A, 5B, and 5C illustrate the outputs of different parts of anexample derivative pixel in accordance with embodiments of thetechnology of the present disclosure.

FIG. 6 illustrates an example double derivative pixel in accordance withembodiments of the technology disclosed herein.

FIG. 7 is a block diagram illustration of an example double derivativepixel in accordance with embodiments of the technology disclosed herein.

FIG. 8 is a block diagram illustration of another example doublederivative pixel in accordance with embodiments of the technologydisclosed herein.

FIG. 9 illustrates an example combined pixel imager in accordance withembodiments of the technology disclosed herein.

FIG. 10 is a block diagram illustration of an example combined pixelimager in accordance with embodiments of the technology disclosedherein.

FIG. 11 illustrates another example combined pixel imager foraccumulating charge in accordance with embodiments of the technologydisclosed herein.

FIG. 12 illustrates another an example simplified combined pixel imagerfor accumulating charge in accordance with embodiments of the technologydisclosed herein.

FIG. 13 is a block diagram illustration of an example simplifiedcombined pixel imager in accordance with embodiments of the technologydisclosed herein.

FIG. 14 illustrates an example direct motion measurement camera inaccordance with the technology of the present disclosure.

FIG. 15 illustrates an example direct motion measurement camera withdithering in accordance with embodiments of the technology disclosedherein.

FIG. 16A illustrates a first frame from a conventional camera in asecurity application.

FIG. 16B illustrates the first frame from a camera in accordance withembodiments of the technology described herein.

FIG. 17A illustrates a second frame from a conventional camera in asecurity application.

FIG. 17B illustrates the second frame as captured by a camera inaccordance with embodiments of the technology disclosed herein

FIG. 18 illustrates an example method of detecting motion within thefield of view of a pixel in accordance with embodiments of thetechnology disclosed herein.

FIG. 19 illustrates an example computing component that may be used inimplementing various features of embodiments of the disclosedtechnology.

FIG. 20 illustrates an example traditional pixel used in current imagesensors.

FIG. 21 is a block diagram illustrating the general approach for afiltering pixel in accordance with various embodiments of the technologydisclosed herein.

FIG. 22 illustrates an example filtering pixel including apseudo-digital filter in accordance with various embodiments of thetechnology disclosed herein.

FIGS. 23A and 23B illustrate example implementations of the filteringpixel described with respect to FIG. 22.

FIG. 24 illustrates an example imaging system implementing an imagesensor with filtering pixels configured with such filtering capability.

FIG. 25A illustrates when the filtering pixel is configured to determinewhen an object is illuminated and remove unwanted light sources.

FIG. 25B illustrates when the filtering pixel is configured to determinewhen an object is moving.

FIG. 25C illustrates a method of switching filter polarity compared tothe modulated illumination.

FIG. 25D illustrates a method of switching filter polarity compared tothe modulated illumination.

FIG. 26 illustrates an example imaging system in accordance with variousembodiments of the technology disclosed herein.

FIG. 27 illustrates an example pixel array in accordance withembodiments of the technology disclosed herein.

FIG. 28 illustrates an example filtering pixel with multiple sample andhold filters in accordance with various embodiments of the technologydisclosed herein.

FIG. 29A illustrates an example of a selected FOV imaging system inaccordance with embodiments of the technology disclosed herein.

FIG. 29B is a block diagram illustrating an example modulation componentthat may be used in the selected FOV imaging system of FIG. 29A.

FIG. 30A is an operational flow diagram illustrating an example methodthat may be implemented by the selected FOV imaging system of FIG. 29A.

FIG. 30B illustrates a representation of an example selected FOV imagethat may be captured in accordance with implementations.

FIG. 30C illustrates a representation of the corresponding image of FIG.30B captured with a standard camera.

FIG. 31A illustrates another example of a selected FOV imaging system inaccordance with embodiments of the technology disclosed herein.

FIG. 31B is a block diagram illustrating an example modulation componentthat may be used in the selected FOV imaging system of FIG. 31A.

FIG. 31C illustrates an example pixel array filter configuration inaccordance with embodiments of the technology disclosed herein.

FIG. 32 is an operational flow diagram illustrating an example methodthat may be implemented by the selected FOV imaging system of FIG. 31A.

FIG. 33A is an operational flow diagram illustrating an example methodthat may be implemented by a 3D imaging system in accordance withembodiments of the technology disclosed herein.

FIG. 33B illustrates a representation of a 3D image that may be capturedfor motion sensing in accordance with one implementation.

FIG. 33C illustrates a representation of the corresponding image of FIG.33B captured with a standard camera.

FIG. 34 illustrates an example implementation of a four-dimensionalstructured light imaging system in accordance with embodiments of thetechnology disclosed herein.

FIG. 35 is an operational flow diagram illustrating an example methodthat may be implemented by the four-dimensional structured light imagingsystem of FIG. 34.

FIG. 36 illustrates an example implementation of a four-dimensionallaser scanning imaging system in accordance with embodiments of thetechnology disclosed herein.

FIG. 37 illustrates an example configuration of a pixel frequency offour filtering pixels as compared to the light source illuminationfrequency that may be implemented by the four-dimensional laser scanningimaging system of FIG. 36.

FIG. 38 illustrates an example implementation of a spectroscopy imagingsystem in accordance with embodiments of the technology disclosedherein.

FIG. 39A illustrates pixel filtering that may be implemented by thespectroscopy imaging system of FIG. 38 based on phase modulation ofdifferent wavebands of light.

FIG. 39B illustrates pixel filtering that may be implemented by thespectroscopy imaging system of FIG. 38 based on phase modulation ofdifferent wavebands of light.

The figures are not intended to be exhaustive or to limit the inventionto the precise form disclosed. It should be understood that theinvention can be practiced with modification and alteration, and thatthe disclosed technology be limited only by the claims and theequivalents thereof.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiments of the technology disclosed herein are directed to usingimage sensors in a variety of different imaging applications. In someimplementations, further described below, the pixels of the image sensormay be configured (e.g., programmed) with one or more filters to sense avariety of things, including, for example: average intensity and color(like a standard camera), 3D depth (like a time of flight or structuredlight camera), changes and/or motion in an image (like an event basedcamera), spectral reflectance (like a spectroscopy camera), and manyother features of an imaged scene or object.

For example, a filtering pixel may be configured with a derivativefilter, a double derivative filter, a sampling filter, a sample and holdfilter, a pseudo-digital filter, a demodulation filter, an electronicdecoding filter, a code division multiplexing filter, some combinationof the aforementioned filters, some combination of the aforementionedfilters and a traditional imaging circuit, or some other type of filter.Example imaging applications in which the disclosed filtering pixeltechnology may be implemented include, for example, selected field ofview (FOV) cameras, three-dimensional cameras, four-dimensionalstructured light cameras, four-dimensional laser scanning cameras,spectroscopy cameras, ambient light insensitive cameras, event basedcameras, time of flight cameras, or some combination thereof. These andother example applications are further described below.

In alternative implementations, further described below, theseapplications may be realized by using digital filters that digitallyfilter the output of an image sensor (e.g., a CMOS sensor), where eachpixel of the image sensor is the same. In such implementations, digitalfiltering may be implemented using a field programmable gate array(FPGA), an application-specific integrated circuit (ASIC), the logic ofthe image sensor itself (after the ADC output), a digital signalprocessor (DSP), a programmable logic circuit (PLC), a graphicalprocessing unit (GPU), a central processing unit (CPU), or some otherdigital processor that may be configured to process the digital outputof an image sensor (e.g., multiple frames captured by the sensor).

Before describing these example applications, it is instructive tocompare the filtering pixels in accordance with the technology disclosedherein with traditional pixels typically utilized in image sensors. Atraditional pixel utilized in image sensors simply outputs the totalintensity of all light falling on the photodetector. FIG. 20 is anillustration of a basic pixel 2000 implemented in image sensor products,such as digital cameras. The pixel 2000 includes a photodiode 2010 forconverting absorbed light into an electric current. The photodiode 2010may be a p-n junction, while other embodiments may utilize a PINjunction (a diode with an undoped intrinsic semiconductor regiondisposed between the p-type semiconductor region and the n-typesemiconductor region). As light falls on the photodiode 2010, thephotodiode 2010 bleeds charge to ground, pulling the output voltage ofthe photodiode 2010 closer to ground. The output voltage of thephotodiode 2010 is an integral of the light intensity on the photodiode2010. Field-effect transistor (FET) 2030, controlled by the drainvoltage V_(dd) 2015, serves a buffer amplifier to isolate thephotodetector 2010. The output from the FET 2030 is proportional to theintegral of the intensity of light on the photodiode 2010. The selectFET 2040 is controlled by the select bus 2025 allows for the output fromthe pixel to be outputted to the read bus 2035. A reset FET 2020 isplaced between the photodiode 2010 and the drain voltage V_(dd) 2015.The reset FET 2020, when activated, serves to make the output voltage ofthe photodiode 2010 equivalent to V_(dd) 2015, thereby clearing anystored charge.

As arranged, each pixel outputs a signal proportional to the integral ofthe intensity of all the light falling on the photodiode 2010. That is,the total intensity of the light falling on the photodiode 2010 isoutputted when the select FET 2040 is selected. That output can bemodified through pre- or post-filtering applied to the image sensor. Forexample, in order to limit the type of light captured by the pixel(i.e., desired wavelengths), optical filters are generally added to theimage sensor over the pixels. In CMOS applications, optical filtersgenerally comprise layers of resist or other material placed above thephotodiode of the pixel, designed to pass only one or more particularwavelengths of light. In some applications, a purpose-built lens ormicrolens array may also be used to filter the type or amount of lightthat reaches each pixel. Such optical filters, however, are not perfect,allowing some stray, undesired light to pass through. Moreover, opticalfilters may be overpowered by high-intensity light sources hindering theeffectiveness of the filter. Accordingly, the output intensity of eachpixel includes such undesired intensity.

Additionally, digital filtering may be applied to the output from eachpixel to filter out certain aspects of the captured light. Traditionaldigital filtering is used to perform complex processing of the capturedlight, including sampling and application of complex algorithms toidentify specific components of the captured light. Before any digitalfiltering may occur, however, the analog signal from the pixel must beconverted into a digital signal, requiring the use of A/D converters.This need to convert the signal into the digital domain limits the speedat which processing may occur. As a result, information in the lightthat is faster than the frame rate of the sensor is lost. Moreover, thedigitized signal includes all of the raw intensity captured by thepixel, meaning that if undesired information was captured it is embeddedwithin the digital signal, limiting the effectiveness of filteringtechniques meant to account for various problems in imaging (e.g.,motion blur).

Accordingly, image sensors are, in essence, broken into three differentdomains: the optical domain, the pixel (analog) domain, and the digitalprocessing domain. Variations of filtering have been applied in theoptical domain and the digital processing domain, designed to filter outor enhance particular aspects of the captured image signal. However, thepixel domain has generally been limited to traditional pixels comprisingmerely a photodetector, or a photodetector and an amplifier in the caseof active pixel sensors. That is, there is no additional filteringconducted in the pixel domain that could assist in overcomingdeficiencies in the optical filtering or to ensure that only desiredimage signals are captured and outputted.

To this end, embodiments of the technology disclosed herein providesfiltering-pixels for use in image sensors. The filtering-pixels of theembodiments disclosed herein provide additional filtering capabilitiesto image sensors, as opposed to the non-filtering characteristics oftraditional pixels. The capabilities of filtering-pixels in accordancewith embodiments of the technology disclosed herein provide an analogfilter within the pixel domain, providing for more efficient filteringand processing of captured light. By adding filtering functionality tothe pixels themselves, various embodiments of the technology allow forgreater differentiation between light sources, enabling more efficientand finer filtering of undesired light sources, reducing saturationcaused by high-intensity light and providing clearer image resolution.Further, greater separation between objects within a scene is possibleby filtering out relevant aspects of the captured light (e.g.,derivatives of the intensity) at the pixel level. That is, filteringpixels in accordance with embodiments of the disclosure herein removeundesired aspects of a captured image such that only the desiredinformation signals are outputted at greater frame rates.

Moreover, various embodiments disclosed herein provide image sensorshaving a plurality of filtering pixels, enabling different informationwithin an image to be captured simultaneously, and the plurality offiltering pixels may be programmable. Moreover, embodiments of thetechnology disclosed herein provide a hybrid analog-digital filter,enabling faster filtering of image signals by removing the need for anA/D converter. Moreover, various embodiments of the technology disclosedherein provide image sensors, cameras employing different embodiments ofthe image sensors, and uses thereof to provide cameras that arecomputationally less intensive and faster than traditional camerasthrough the use of filtering pixels.

FIG. 21 is a block diagram illustrating the general approach for afiltering pixel 2100 in accordance with various embodiments of thetechnology disclosed herein. As illustrated, the photodiode 2120 of thefiltering pixel 2100 absorbs photons from light, outputting a currentproportional to the intensity of the light (like a traditional pixel). Afilter 2130 is embedded in the filtering pixel 2100. As will beexplained in greater detail below, the filter 2130 may be configured toperform one of several different filtering functions. For example, thefilter 2130 may filter to pass one or more derivatives of the outputsignal from the photodiode 2120. In other embodiments, the filter 2130may perform a pseudo-digital sampling of the analog signal, adding andsubtracting these samples to filter out particular aspects of the outputsignal from the photodiode 2120. The integrator 2140 outputs anintegrated version of the signal output from the filter 2130. The outputof the integrator 2140 would be digitized by an analog to digitalconverter.

Although described with respect to integrating within the currentdomain, a person of ordinary skill in the art would understand that thefiltering approach described with respect to FIG. 21 may be done in thevoltage domain. That is, instead of performing the filtering on acurrent signal, the output from the photodiode 2120 could be convertedinto a voltage and the filtering applied to the voltage. In otherembodiments, portions of the filtering may be applied to a currentsignal while others are applied after the current signal is convertedinto a voltage signal. A person of ordinary skill in the art wouldunderstand how to modify the circuitry of the filtering pixel 2100 inenable filtering to occur in the current or voltage domains.

As alluded to above, filtering pixels in accordance with embodiments ofthe technology disclosed herein can be equipped with a variety offiltering capabilities. This enables a range of various signal aspectsto be outputted by the filtering pixel, increasing the type ofinformation about the scene that can be identified with higherresolution. For example, in various embodiments, the filtering pixelsmay filter to pass aspects of the signal indicative of motion within acaptured scene. The conventional frame subtraction technique formeasuring the displacement of objects, which can be interpreted asmovement, within the FOV of a camera suffers from several disadvantages.One such disadvantage is a low update rate. For example, if two framesare subtracted to determine the motion of an object, displaying theresulting image requires the time needed to capture two frames and thecomputation time required to subtract the two frames. In addition, sinceeach frame is an integral of the intensity for every pixel during theexposure time, changes that take place during each frame time aremissed. Moreover, the frame subtraction technique is sensitive to blureffects on images, caused either by other objects within the camera'sFOV or the motion of the camera itself. Changes in illumination alsoimpact the ability to rely on frame subtraction, as it is more difficultto distinguish motion where the light reflecting off of or throughobjects varies independent of the motion of the objects. In summary,traditional cameras and image sensors are designed to capture non-movingimages. As soon as there is motion or change in the scene, image qualityis compromised and information is lost. Cameras and image sensorsimplementing embodiments of the technology disclosed herein provideadvantages over traditional cameras and image sensors for a variety ofdifferent applications. Background separation is easier as only motionis captured through the use of derivative sensing. Conventional camerascapture all objects within a frame. This makes it more difficult toseparate out moving objects from stationary objects in the background.For security applications, by limiting the amount of informationcaptured it is easier to identify objects of interest within a scene.

Further, frame subtraction is computationally intensive. Conventionalimage sensors in cameras generally capture all objects within the FOV ofthe camera, whether or not the object is moving. This increases theamount of data for processing, makes background separation moredifficult, and requires greater processing to identify and measuremotion within the FOV. This renders motion detection through framesubtraction computationally intensive.

By utilizing a filtering pixel configured to filter one or morederivatives of the light intensity in accordance with embodiments of thetechnology disclosed herein, less computationally intensive and fastermotion detection is provided compared with cameras employing traditionalpixels. Traditional image sensors integrate the light that falls on eachpixel and output how much light fell on that pixel during the frame. Incontrast, image sensors in accordance with various embodiments describedherein can detect changes in the light falling on pixels of the imagesensor during the frame. In particular, the image sensors describedherein detect the change in the intensity of light detected on thephotodiodes of the pixels comprising image sensors, rather thandetecting mere intensity on that photodiode (as in traditional imagesensors). Where the intensity is constant (i.e., no movement), novoltage is accumulated. However, if the intensity of the light isvarying (i.e., there is movement within the FOV), the output will beproportional to the number and size of changes during the frame. Themore changes, or the larger the change, the larger the output voltage.In some embodiments, this type of image sensor employing derivativesensing may be implemented within a conventional camera to directlydetect movement in addition to capturing the entire scene.

For example, FIG. 16A illustrates a frame captured using a conventionalcamera in a security application. As illustrated, all objects within theFOV of the camera are captured, including the ground, the car, and anindividual that is walking. By capturing each of these objects, a largeamount of information is obtained and stored when all that is ofinterest may be the movements of the individual in the scene. Incontrast, FIG. 16B illustrates a frame captured by a camera implementingembodiments of the technology described herein. As can be seen in FIG.16B, only the man (who is walking to the bottom left corner of theframe) is captured. By not capturing all the elements in the background,identification of a moving object is made easier.

In some embodiments, the camera in FIG. 16B may capture not only motionwithin the frame, but also capture and provide quantitative informationon the motion (e.g., the velocity of the moving object; acceleration ofthe object; direction of motion; etc.). This information may be conveyedin a number of different ways. Some non-limiting examples may include:different colors indicating different velocities or ranges ofvelocities; textual layovers; graphical indications; or a combinationthereof. Post-processing may be utilized to augment the derivativesensed images captured by the camera. Because the information isdirectly sensed through the derivative sensing capability of embodimentsof the technology disclosed, post-processing of the frame is lesscomputational intensive than is required to obtain the same informationusing traditional cameras. In some embodiments, the quantitative imagefor each derivative pixel is used as the input of a control system. Forexample, in some applications, faster moving objects may be moreimportant. In some embodiments, the derivative pixel image may befurther processed or filtered to eliminate noise or false images.

Implementations of embodiments of the technology disclosed herein mayalso provide benefits for other applications, such as gesturerecognition, obstacle avoidance, and target tracking, to name a few.FIG. 17A shows a frame captured with a conventional camera, where anindividual is making a “come here” gesture, as if calling a dog. Similarto the camera in FIG. 16A, all of the elements in the background of thescene including the object of interest (in this case, the gesturingindividual) are captured by the camera of FIG. 17A, making backgroundseparation difficult. Moreover, the gesture being made is moving awayfrom the camera, which makes identification of the gesture moredifficult because the motion is perpendicular to the image sensor. Thisis especially true for slight motions away from the camera, which maynot be visibly noticeable. FIG. 17B illustrates the same frame ascaptured by a camera implementing embodiments of the technologydisclosed herein. As illustrated in FIG. 17B, only the gesture iscaptured by the camera. Moreover, the captured image illustrates agradient representation, whereby the portion of the gesture with whichgreater is motion associated (the upper portions of the fingers) isrepresented by a more pronounced shade of white. This is because, as thefingertips move more than the knuckles, more charge is accumulatedwithin the derivative sensing circuit (as will be discussed in greaterdetail below).

Such implementations also make it easier to identify and detect lowenergy motions. Conventional cameras and time of flight cameras thatcapture the entire scene require large motions that are easilydistinguishable from the background in order to identify gestures ormoving objects. By implementing embodiments of the technology disclosedherein, it is possible to capture low energy motion more easily,allowing for greater motion detection and identification. Even slightmotions, like a tapping finger, can be picked up (which could otherwisebe missed using traditional image sensors).

Various embodiments of the technology disclosed herein utilizederivative filtering to directly detect and capture motion within thescene. FIG. 1 is a circuit diagram of an example filtering pixel 100configured to filter out the derivative of the intensity of light, inaccordance with embodiments of the technology disclosed herein. In theillustrated example, the filtering pixel 100 is a complementarymetal-oxide-semiconductor (CMOS), which captures images by scanningacross rows of pixels, each with its own additional processing circuitrythat enables high parallelism in image capture. The example filteringpixel 100 may be implemented in place of traditional pixels in anactive-pixel sensor (APS) for use in a wide range of imagingapplications, including cameras. As illustrated in FIG. 1, the filteringpixel 100 includes a photodiode 101. The photodiode 101 operatessimilarly to the photodiode 2010 discussed above with respect to FIG.20.

Resistor 103 converts the current generated by the photodiode 101 into avoltage. In some embodiments, the resistor 103, and any other resistorsillustrated throughout, may be implemented by using a switchedcapacitor. Capacitor 104 and resistor 105 act as a high-pass filter.Capacitor 104 serves to filter the alternating current (AC) componentsof the input voltage (when the intensity of the light on the photodiode101 varies or changes), allowing an AC current to pass through. Aderivative of the input voltage is taken by said capacitor 104 andresistor 105, which serves to connect to the supply and eliminate any DCvoltage component. The derivative signal is integrated by thefield-effect transistor (FET) 106 and capacitor 107 (i.e., anintegrator), to generate the output voltage of the filtering pixel 100.The capacitor 104 and resistor 105 forming the high pass filter and maybe referred to as a “differentiator,” a type of circuit whose output isapproximately directly proportional to the time derivative of the input.A second FET 108 reads out the output voltage when the Select Bus 109voltage (i.e., gate voltage for second FET 108) is selected to “open”the second FET 108 (i.e., a “conductivity channel” is created orinfluenced by the voltage).

When the intensity of light falling on the photodiode 101 is constant,there is a constant amount of current flowing through the photodiode 101and the resistor 103 from the voltage V_(dd) 102 into ground. Thiscreates a constant voltage at the input of the capacitor 104 (and nosignal goes through capacitor 104). When the intensity of light fallingon the photodiode 101 varies, an AC component of the signal is createdthat is coupled through the capacitor 104. This AC component of thesignal is integrated by the FET 106 and capacitor 107, and theintegrated signal is read out on the Read Bus 110 when the row (withinan APS) containing the pixel is selected through a proper voltage beingapplied to the Select Bus 109.

In some cases, the amount of light falling on the photodiode may be verysmall. For such low intensity changes, it may be beneficial to include acharge accumulator and/or buffer before the differentiator. FIG. 2illustrates another example filtering pixel 200 configured to filter topass the derivative of the intensity of light in accordance withembodiments of the technology disclosed herein. Similar to the examplefiltering pixel 100 of FIG. 1, the example filtering pixel 200 may beimplemented in place of traditional pixel sensors in an array of anactive-pixel sensor (APS) for use in a wide range of imagingapplications, including cameras. In various embodiments, the photodiode201 may use a p-n junction, while other embodiments may utilize a PINjunction (a diode with an undoped intrinsic semiconductor regiondisposed between the p-type semiconductor region and the n-typesemiconductor region). As light falls on the photodiode 201, thephotodiode 201 bleeds charge to ground, pulling the output voltage ofthe photodiode 201 closer to ground. The output voltage of thephotodiode 201 is an integral of the light intensity on the photodiode201. A reset FET 203 is placed between the photodiode 201 and the drainvoltage V_(dd) 202. The reset FET 203, when activated, serves to makethe output voltage of the photodiode 201 equivalent to V_(dd) 202,thereby clearing any stored charge. In various embodiments, a pixelreset voltage (not pictured) may be utilized as the gate voltage of thereset FET 203. This can be utilized in between frames, so that minimalresidual charge carries over from frame to frame. In some embodiments, acontrolled leakage path to V_(dd) may be used to counteract the chargeaccumulation, as was done by resistor 103 in the embodiment described inFIG. 1. In some embodiments, this controlled leakage is provided bypulsing or otherwise controlling the leakage through FET 203.

In various embodiments, a buffer FET 204 may be included. When theamount of light falling on the photodiode 201 varies, the buffer FET 204enables reading out of the output voltage of the photodiode 201 (whichwill accumulate with changes in the amount of light) without taking awaythe accumulated charge. As illustrated, the gate voltage of the bufferFET 204 is the voltage from the photodiode 201, and both the buffer FET204 and the photodiode 201 have a common drain (in this case, V_(dd)202). The output voltage of the buffer FET 204 is proportional to theintegral of the intensity of light on the photodiode 201. The buffer FET204 also serves to isolate the photodiode 201 from the rest of theelectronic circuits in the pixel.

In various embodiments, a first differentiator 205 takes a derivative ofthe output voltage of the buffer FET 204. The first differentiator 205includes capacitor 206, resistor 207, and FET 208. The output voltage ofthe first differentiator 205 is proportional to the instantaneousintensity on the photodiode 201.

To identify motion within the FOV of the filtering pixel 200 (and toreverse the integration of the accumulator (e.g., FET 204)) the changein intensity of the photodiode is determined. A second differentiator209 takes the derivative of the output voltage of the firstdifferentiator 205. The output voltage of the second differentiator 209is proportional to the change in intensity on the photodiode 201. Thecapacitor 213 integrates the output voltage of the second differentiator209. When the Select Bus 215 is set to open the FET 214, the integratedoutput voltage is read out on the Read Bus 216.

FIG. 3 is a block diagram illustrating an example filtering pixel 300 inaccordance with embodiments of the technology disclosed herein. Thefiltering pixel 300 may be one embodiment of the filtering pixel 200discussed with respect to FIG. 2. As illustrated in FIG. 3, thefiltering pixel 300 includes a photodiode 301. The photodiode 301 isconnected to a storage circuit 302 for storing the charge from thephotodiode 301 and converting to a voltage. As illustrated in FIG. 1,the storage circuit is optional in some embodiments and the signal fromthe photodiode may go directly to the derivative circuit. In someembodiments, the storage circuit 302 may include a buffer FET, similarto the buffer FET 204 discussed with respect to FIG. 2. Variousembodiments may also include a reset FET for clearing any stored chargein between frames (utilizing the pixel reset voltage 304 to control thegate of the reset FET), similar to the reset FET 203 and discussed withrespect to FIG. 2.

The output of the storage circuit 302 goes through the derivativecircuit 303. In various embodiments, the derivative circuit 303 may besimilar to the first and second differentiators 205, 209 discussed withrespect to FIG. 2. The derivative circuit 303 outputs an output voltageproportional to the change in intensity on the photodiode 301.

The output of the derivative circuit 303 is integrated and read outthrough the read out circuit 305 when the select bus 306 voltage is setto activate the read out circuit 305. The read out circuit 305 may besimilar to the capacitor 213 and FET 214, discussed with respect to FIG.2. The output of the read out circuit 305 is the number and size ofchanges in intensity on the photodiode 301 during the frame, and is readout on the read bus 307.

In some embodiments, the derivative circuit may be placed before thestorage circuit, thereby enabling storage of changes during the frame sothey may be stored and read out using standard select bus and read busarchitecture. FIG. 4 illustrates an example of such a filtering pixel400 in accordance with embodiments of the technology disclosed herein.The components of the filtering pixel 400 are similar to those discussedwith respect to FIG. 3, except that the derivative circuit 402 is placedin front of the storage circuit 403.

FIGS. 5A, 5B, and 5C illustrate the output of a filtering pixeldescribed according to FIG. 1. FIG. 5A illustrates the input voltageV_(in), which is proportional to the intensity of the light falling onthe photodiode, as a function of time. FIG. 5B illustrates the outputvoltage of the derivative circuit V_(out) (derivative). FIG. 5Cillustrates the corresponding integrated output voltage V_(out)(integrated) through the derivative pixel as a function of time. It canbe appreciated that as V_(in) fluctuates, the output of the derivativecircuit (FIG. 5B) changes in steps corresponding to the rate of changein the input voltage. In bipolar circuits, the output voltage of thederivative can be negative in some cases (when the input voltage drops).In some embodiments, a unipolar CMOS circuit may be used, which wouldpeg the output voltage of the derivative at zero (eliminating thenegative voltage portion of FIG. 5B). As illustrated in FIG. 5C, theintegrated output voltage accumulates as the input voltage changes. Insome embodiments, the circuit is designed so that the positivederivative is eliminated and the negative derivative is stored. Forexample, the FET 106 may be replaced by a circuit that inverts thesignal. In some embodiments, one portion of the circuit outputs thestored positive derivative and another portion of the circuit outputsthe stored negative derivative.

APSs and other imaging sensors implementing filtering pixels inaccordance with the embodiments discussed with respect to FIGS. 1-4enable direct measurement of motion within the FOV of the imagingsensor, without the need for frame subtraction. The motion is measureddirectly, instead of deriving the motion through post processing. Thisprovides a visual representation of movement within the frame. By addinganother derivative circuit to the filtering pixel, not only can thechange in intensity of the light be measured, but also the change in thechange of intensity of light may be measured. That is, the singlederivative circuit (as that described with respect to FIGS. 1-4) sensesvelocity in the image, while the double derivative circuit sensesacceleration within the image. By including an additional derivativecircuit to create a double derivative circuit in the filtering pixel,the quality of the motion within the frame may be measured directly, aswell as the motion itself, without the need for frame subtraction orother techniques. In some embodiments, even higher order derivatives aretaken to measure jerk, snap, etc.

FIG. 6 illustrates an example filtering pixel 600 with a doublederivative circuit in accordance with embodiments of the technologydisclosed herein. The filtering pixel 600 includes the same componentsas the filtering pixel 200 discussed with respect to FIG. 2. For ease ofdiscussion, similar components in FIG. 6 are referenced using the samenumerals as used in FIG. 2, and should be understood to function in asimilar manner as discussed above. Referring to FIG. 6, the filteringpixel 600 adds an additional differentiator 610 to the circuit, whichtakes the derivative of the derivative resulting from the differentiator209. The output voltage of the additional differentiator 510 isintegrated by capacitor 213 and read out on the read bus 216 by the FET214. In some embodiments, the output of the differentiator 209 may alsobe read out as well by connecting a second capacitor and FET after theFET 212, and connecting the new FET to the select bus 215 and a secondread bus.

Although described with respect to single and double derivatives,nothing in this disclosure should be interpreted to limit the number ofderivative circuits that may be utilized. Additional higher orderderivatives (e.g., triple derivative) may be created by adding moredifferentiators to the filtering pixel. Accordingly, a person ofordinary skill would appreciate that this specification contemplatesderivative pixels for higher order derivatives.

FIG. 7 is a block diagram illustration of an example filtering pixel 700with a double derivative circuit in accordance with embodiments of thetechnology disclosed herein. The filtering pixel 300 may be similar tothe filtering pixel 600 discussed with respect to FIG. 6. For ease ofdiscussion, similar components in FIG. 7 are referenced using the samenumerals as used in FIG. 3, and should be understood to function in asimilar manner as discussed above. In place of the derivative circuit303, the filtering pixel 700 includes a double derivative circuit 703.The double derivative circuit 703 may be similar to the first, second,and third differentiators 205, 209, 610 discussed with respect to FIG.6. The double derivative circuit 703 outputs an output voltageproportional to the change in the change of intensity on the photodiode301. In some embodiments, the read out circuit 305 may includeadditional circuitry to read out the output voltage from eachdifferentiator stage in the double derivative circuit 703. In suchembodiments, the filtering pixel 700 may include an additional read bus.

In some embodiments, the derivative circuit may be placed before thestorage circuit, thereby enabling storage of changes during the framemay be stored and read out using standard select bus and read busarchitecture. FIG. 8 illustrates an example of such a filtering pixel800 in accordance with embodiments of the technology disclosed herein.The components of the filtering pixel 800 are similar to those discussedwith respect to FIGS. 4 and 7, except that the double derivative circuit802 is placed in front of the storage circuit 403. In some embodiments,a single derivative circuit may be placed before the storage circuit anda second derivative circuit placed after the storage circuit. Othervariations of the order of derivative and storage circuits exist. Theremay be multiple derivative circuits and or multiple storage circuitsarranged in different orders.

Image sensors (or cameras) containing arrays of filtering pixels similarto the embodiments discussed with respect to FIGS. 1-8 allow for directmeasurement of motion within a frame. In essence, such image sensorsdetect changes in the light falling on each pixel, and the read out ofthe pixels represent the motion occurring within the frame. This enablesmotion measurement without the need for intensive processing techniqueslike frame subtraction.

In various embodiments, the derivative sensing may be combined withtraditional image capturing circuitry. FIG. 9 illustrates an examplecombined pixel imager 900 in accordance with embodiments of thetechnology disclosed herein. The example combined pixel imager 900 addsseveral components to the filtering pixel 100 discussed with respect toFIG. 1. For ease of discussion, similar components in FIG. 9 arereferenced using the same numerals as used in FIG. 1, and should beunderstood to function in a similar manner as discussed above. Referringto FIG. 9, in addition to the filtering pixel components, resistor 901and capacitor 902 serve as an low pass filter of the current from thephotodiode 101 to capture the total amount of light falling on thephotodiode during the frame. In various embodiments, FET 903 serves as abuffer to enable reading the integrated signal from the capacitor 902and resistor 901. When the signal is read out on the traditional signalbus 905 by activating the FET 904 with the select bus 109 voltage, theoutput signal (when combined with the read outs from the other pixels inthe sensor) results in a traditional image being captured. At the sametime, the derivative output may be read out on the derivative signal bus910 (similar to the read bus 110 of FIG. 1). In this way, both atraditional image, as well as the derivative signal, may be obtainedwithin the same combine pixel imager 900 package.

FIG. 10 is a block diagram illustration of an example combined pixelimager 1000 in accordance with embodiments of the technology disclosedherein. The illustrated example combined pixel imager 1000 is a blockdiagram representation of the combined pixel imager 900 discussed withrespect to FIG. 9. Referring to FIG. 10, the combined pixel imager 1000includes common components to the filtering pixel 400 discussed withrespect to FIG. 4. In this way, the derivative may be integrated(through 402, 403, 405) so that changes during a frame can be stored andread out using standard select bus and read bus (derivative signal bus1010) architecture. At the same time, the integration circuit 1002integrates the current of the photodiode 401 to generate a traditionalread out signal (i.e., capturing the entire scene within the FOV of thecombined pixel imager 1000). In various embodiments, the integrationcircuit 1002 contains the components comprising the integrator in FIG. 9(i.e., resistor 901, capacitor 902). The storage circuit 1003 and readout circuit 1005 operate in a manner similar to the storage circuit 403and read out circuit 405, with the read out circuit 1005 connected tothe traditional signal bus 1020 instead of the derivative signal bus1010.

In some cases, the amount of light falling on the photodiode may be verysmall. For such low intensity changes, it may be beneficial to include acharge accumulator and buffer before the differentiator, similar to theexample filtering pixel 200 discussed with respect to FIG. 2. FIG. 11illustrates another example combined pixel imager 1100 for accumulatingcharge in accordance with embodiments of the technology disclosedherein. For ease of discussion, similar components in FIG. 11 arereferenced using the same numerals as used in FIGS. 2 and 9, and shouldbe understood to function in a similar manner as discussed above.Inclusion of the buffer FET 204 isolates the photodiode 201, enablingfor the additional capacitor 902 and resistor 902 to be removed from thecircuit illustrated in FIG. 9. This further renders the FET 903redundant. In various embodiments, the FET 903 may be removed (anexample simplified combined pixel imager 1200 is illustrated in FIG.12). Additional differentiators may be included in some embodiments tocreate a higher order derivative circuit (e.g., double derivative;triple derivative), in a manner similar to the example double derivativepixel 600 discussed with respect to FIG. 6.

When the redundant FET is removed, the combined pixel imager can besimplified. FIG. 13 is a block diagram illustration of the examplesimplified combined pixel imager 1300 in accordance with embodiments ofthe technology disclosed herein. As illustrated in FIG. 13, thesimplified combined pixel imager 1300 may include similar components asdiscussed with respect to FIG. 3. Accordingly, for ease of discussion,similar components in FIG. 13 are referenced using the same numerals asused in FIG. 3, and should be understood to function in a similar manneras discussed above. Unlike the combined pixel imager 1000 of FIG. 10,the simplified combined pixel imager 1300 can provide dual read out ofthe traditional signal as well as the derivative signal with only theaddition of a second read out circuit 1305 for the traditional imagingfunction.

Traditional CMOS APSs contain an array of pixels, and have beenimplemented in many different devices, such as cell phone cameras, webcameras, DSLRs, and other digital cameras and imaging systems. Thepixels within this array capture only the total amount of light fallingon the photodiode (photodetector) of the pixel, necessitatingcomputationally intensive frame subtraction and other techniques inorder to identify motion within the images. By replacing the array oftraditional pixels with an array of filtering pixels as those discussedwith respect to FIGS. 1-13, it is possible to capture the motionoccurring within a frame, in lieu of or in addition to capturing theentire frame (like a traditional pixel).

The embodiments discussed with respect to FIGS. 1-13 enable measurementof motion within the frame that is in-plane, i.e. parallel to thesurface of the image sensor. To provide measurement of out-of-planemotion (i.e., motion towards and away from the image sensor), anillumination of the scene can be used to convert out-of-plane motion tovariations in intensity, which the filtering image sensor previouslydescribed can detect. For example, a structured light illumination wheresome sections are illuminated and others are dark can create intensityfluctuations on the object, and these fluctuations in intensity can bedetected with the filtering pixel technology. In such embodiments, aknown pattern of light (e.g., grids) is projected onto the scene and,based on the deformation caused by striking the surface as an object inthe scene moves towards or away from the camera, the rate of motion maybe calculated. In various embodiments, imperceptible structured light(e.g., infrared) may be utilized as to not impair the visual capture ofthe entire scene.

In some embodiments, a coherent laser radar component may be added tothe implementation. FIG. 14 illustrates an example direct motionmeasurement camera 1400 in accordance with the technology of the presentdisclosure. The direct motion measurement camera 1400 includes an imagesensor 1401. The image sensor 1401 includes an array of filtering pixelssimilar to the filtering pixels discussed above with respect to FIGS.1-5 in some embodiments. In other embodiments, the image sensor 1401includes an array of higher order filtering pixels similar to the higherorder filtering pixels discussed with respect to FIGS. 6-8. The imagesensor 1401 may include a combined pixel imager similar to the combinedpixel imager discussed with respect to FIGS. 9-13, in variousembodiments. The direct motion measurement camera 1400 further includesan imaging system, comprising a lens 1402 and an aperture stop 1403. Theimaging system serves to ensure that light is directed correctly to fallon the image sensor 1401. In various embodiments, the direct motionmeasurement camera 1400 may include filtering pixels discussed withrespect to FIGS. 21-23.

To provide the out-of-plane motion measurement, the direct motionmeasurement camera 1400 takes advantage of the Doppler shift. A laserdiode 1404 is disposed on the surface of the aperture stop 1403. Thelaser diode 1404 emits a laser towards a prism 1405. The light emittedfrom the laser diode 1404 is divergent, and the divergence angle of thelaser may be set by a lens (not shown) of the laser diode 1404. Invarious embodiments, the divergence angle may be set to match thediagonal FOV of the camera 1400. Polarization of the light emitted fromthe laser diode 1404 may be selected to avoid signal loss frompolarization scattering on reflection from the moving object 1406. Insome embodiments, the polarization may be circular; in otherembodiments, linear polarization may be used. To eliminate specklefading, the pixel size of the image sensor may be matched to the airydisk diameter of the imaging system in various embodiments.

The prism 1405 has a first power splitting surface (the surface closestto the laser diode 1404), which reflects the light 1410 emitted by thelaser diode 1404 outwards away from the camera. In addition, a portionof the light emitted from the laser diode 1404 passes through the firstsurface of the prism and reflects 1420 off of a second, interior surfaceof the prism 1405 towards the lens 1402 of the imaging system, throughan opening 1407 in aperture stop 1403. The prism 1405 is designed as aband pass filter corresponding to the wavelength of the light emittedfrom the laser diode 1404. Accordingly, the prism 1405 is opaque (likethe aperture stop 1403) to visible light.

As can be seen in FIG. 14, the light 1410 that is transmitted outwardsaway from the camera 1400 reflects off of a moving object 1406 backtowards the camera 1400 (as illustrated at 1430). The lens 1402 capturesthe returning light 1430 through the aperture stop 1403, and focuses thereturning light 1430 onto the pixel on the image sensor 1401corresponding to the location of the moving object 1406.

The returning light 1430 off of the moving object 1406 is shifted infrequency by the Doppler shift. If the moving object 1406 was stationaryinstead, no shift in the frequency would occur. For the moving object1406, however, the shift in frequency is proportional to the velocity ofthe moving object 1406. Accordingly, the reflected light 1420 and thereturning light 1430 each have a different frequency. When the twolights combine on the image sensor 1401, a beat frequency is generatedthat is equal to the difference in the frequencies between the reflectedlight 1420 and the returning light 1430. By measuring this beatfrequency, the direct motion measurement camera 1400 may measure motiontowards or away from the camera without the need for additionalpost-processing. The filtering pixel sensor previously described candetect the beat frequency since the derivative of a sine wave is acosine wave. Thus, the first derivative sensing circuit will sense thesame as the double derivative circuit. Also, the positive derivative andthe negative derivative circuits will also detect the same signal. Thisis different than any other intensity variation that the pixel maydetect, related to lateral motion of the object.

In some embodiments, not only can the velocity of motion in theout-of-plane direction be measured, but the direction of theout-of-plane motion may also be measured. By dithering the prism, thefrequency of the outgoing light (1410 in FIG. 14) can be changedrelative to the reflected light (1420 in FIG. 14). By doing this, thedirection of motion of the moving object may be detected. Moreover, thebeat frequency can be further distinguished from the object's in-planemotion. FIG. 15 illustrates an example direct motion measurement camerawith dithering 1500 in accordance with embodiments of the technologydisclosed herein. The direct motion measurement camera with dithering1500 has a similar configuration to the direct motion measurement camera1400 discussed with respect to FIG. 14. Referring to FIG. 15, the directmotion measurement camera with dithering 1500 includes a lens 1510,aperture stop 1520, laser diode 1530, prism 1540, an actuator 1550, anda housing 1560. Although not pictured, an image sensor is disposedbehind the lens 1510 and within the housing 1560, and may be similar tothe image sensor discussed with respect to FIG. 14. The prism 1540 ispositioned over the opening in the aperture stop 1520 (the opening isshown by the cross-hatched rectangle in the prism 1540).

To provide the dither, the prism 1540 is connected to an actuator 1550.In various embodiments, the actuator 1540 may be amicroelectromechanical system (MEMS) actuator, such as the actuatordisclosed in U.S. patent application Ser. No. 15/133,142. The actuator1540 may be configured such that the prism 1540 adds to Doppler shiftduring even frames, but subtracts from Doppler shift during odd frames.When Doppler shift is positive (i.e., objects are moving towards thecamera), even numbered frames will have a larger Doppler shift than oddnumbered frames. When Doppler shift is negative (i.e., objects aremoving away from the camera), even numbered frames will have a smallerDoppler shift than odd numbered frames.

Although discussed with respect to embodiments employing coherent light,the dithering element may be utilized with other forms of illuminationas well. For example, embodiments of the technology disclosed hereinutilizing structured light may include the dithering effect to takeadvantage of Doppler shift. Nothing in this specification should beinterpreted as limiting the use of the prism and dithering componentdescribed to a single illumination technique. A person of ordinary skillin the art would appreciate that the dithering technique is applicablewherever the Doppler shift could be utilized to identify out-of-planemotion of objects within the FOV of the derivative sensor.

As discussed above, cameras implementing the technology discussed hereinprovide advantages over traditional cameras. Frame subtraction is notneeded in order to determine motion within the frame. The derivativecircuits and image sensors discussed above with respect to FIGS. 1-15allow for direct measurement of motion of object within a frame. Onlythe number and size of the motion is captured through the derivativecircuit, making it easier to separate a moving object from thebackground and identifying the moving object. By combining thederivative sensing technology with a traditional pixel circuit, it ispossible to both collect visual images of the entire scene (like aconventional camera) and also capture only the motion within the scene,at the same time. The ability to capture the motion of objects throughthe scene is useful for many different types of applications, includingthe security, gesture recognition, obstacle avoidance, and targettracking applications discussed, among others. The implementationsenable only the motion in the scene to be captured, making it easier todifferentiate between the background elements and the moving object.

As an example, a camera in accordance with embodiments of the technologydisclosed herein may be built for use in cellular devices. In oneembodiment, an aperture size of 2.5 mm diameter and an F number of 2.8(traditional size of a camera in a cellular device) can be assumed. AFOV can be 74.5 degrees. To avoid speckle fading, the filtering pixelsize may be set to be smaller than 6.59 μm (dependent on the aperturesize and F#). The camera resolution can be assumed to be 1.25megapixels, or 0.94 mm lateral resolution at 1 m distance. Also, a laserdiode power of 100 mW can be assumed. If there is an object moving at 1m away from the camera, about 90 mW of the laser diode power may be usedto illuminate the object at this distance, and an illuminationequivalent to about 2 lux in the visible (1 lux is 1.46 mW/m̂2 at 555 nm)can be assumed. Assuming 30% diffuse object reflectivity, the powerreceived per pixel is 7e-15 W or 354 photons per frame running at 100fps. A high pass filter can be placed on each filtering pixel that onlydetects between 1 kHz and 100 MHz frequencies. Thus, object motion canbe between 0.49 mm/sec (1.8 m/hr) and 49 m/sec (176 km/hr) can bedetected. In this arrangement, there are over 16 bits of dynamic range.

FIG. 18 illustrates an example method of detecting motion within thefield of view of a pixel. At 1810, a change in intensity of lightfalling on a photodiode of a pixel is sensed. This sensing may besimilar to the sensing discussed above with respect to the photodiode ofFIGS. 1-15. As the intensity of light falling on the photodiode changes,the current flowing out of the photodiode to the drain voltage (V_(dd))varies, varying the voltage of the signal. This change in intensitygenerates an AC component to the output voltage (signal). In someembodiments, this may be sensed through a resistor, such as resistor 103discussed with respect to FIG. 1. In such embodiments, the signal may beproportional to the intensity of the light falling on the photodiode. Invarious embodiments, the change in intensity may be sensed through asource follower transistor, such as the buffer FET 204 discussed abovewith respect to FIG. 2. In such embodiments, the output of the sourcefollower is proportional to the integral of the intensity of the lightfalling on the photodiode.

At 1820, a differentiator takes a derivative of the voltage. Thederivative of the output signal from 1810 is the change in intensity ofthe light falling on the photodiode. In various embodiments, thedifferentiator may comprise a capacitor, resistor, and a FET. In someembodiments, additional derivatives of the signal may be taken tomeasure different aspects of the signal. In embodiments where a sourcefollower is utilized (such as, for example, the buffer FET 204 discussedabove with respect to FIG. 2), two differentiators may be used to obtainthe first derivative of the signal. As discussed above with respect toFIG. 2, in such embodiments the output from the source follower (bufferFET) is proportional to the integral of the intensity of light fallingon the photodiode. Accordingly, the signal from 1810 must first beprocessed to obtain a signal proportional to the intensity of the lightfalling on the photodiode, and then the first derivative may be taken.

When applicable, additional derivatives may be taken at 1830. Additionalderivatives are possible by including an additional differentiator inthe circuit prior to integration (represented by the nth derivative inFIG. 18). As additional higher order derivatives are an option, 1830 isillustrated as a dashed box. If no higher order derivatives are taken,the method would go from 1820 to 1840.

At 1840, the derivative signal is integrated. In various embodiments, acapacitor may be used to integrate the derivative signal. At 1850, theintegrated signal is read out on a read bus. In various embodiments, arow select FET may be utilized to read out the integrated signal.

In addition to filtering out derivatives of the intensity of light,filtering pixels in accordance with embodiments of the technologydisclosed herein may include a sampling filter. In signal processing,sampling filters are digital filters that operate on discrete samples ofan output signal from an analog circuit (e.g., a pixel). As it is in thedigital domain, sampling requires that the analog output signal from thecircuit is converted into a digital signal through an A/D converter. Amicroprocessor applies one or more mathematical operations making up thesampling filter to the discrete time sample of the digitized signal.However, such processing is time- and resource-intensive. Moreover, theeffectiveness of the process is limited by the quality of the analogsignal outputted from the circuit.

By including a sampling filter within the pixel itself, the usefulnessof sampling filters used in the digital domain can be realized in theanalog domain of the pixel circuit. In other words, various embodimentsof the technology disclosed herein provide filtering pixels providing ahybrid analog-digital filtering capability within the pixel itself. Byincluding a pseudo-digital filter circuit, such filtering pixels canperform digital-like sampling of the analog signal, without the need foran A/D converter.

For example, filtering pixels including a pseudo-digital filter mayfilter out undesired light sources. This capability can be used, forexample, to address light saturation caused by high-intensity lightsources (e.g., the sun). In some embodiments, this capability can beused to improve visibility in certain conditions, e.g., foggyconditions. That is, filtering pixels can be used to better identifymoving objects through the fog. In some embodiments selective lightingfiltering can be used to allow viewing only those objects illuminated bya particular code. As discussed above, traditional pixels used in imagesensors integrate the total intensity of light falling on its photodiodeduring a frame. That is, each pixel can be thought of as a well,accumulating captured light corresponding to an increase in the totalintensity of light captured. As with physical wells, however, the amountof light that may be captured by each pixel is limited by its size. Whena pixel approaches its saturation limit, it loses the ability toaccommodate any additional charge (caused by light falling on thephotodiode). In traditional cameras, this results in the excess chargefrom a saturated pixel to spread to neighboring pixels, either causingthose pixels to saturate or causing errors in the intensity outputted bythose neighboring pixels. Moreover, a saturated pixel contains lessinformation about the scene being captured due to the maximum signallevel being reached. After that point, additional information cannot becaptured as the pixel cannot accommodate anymore intensity.

FIG. 22 illustrates an example filtering pixel 2200 including apseudo-digital filter in accordance with various embodiments of thetechnology disclosed herein. The example filtering pixel 2200 is similarto the filtering pixel 200 discussed with respect to FIG. 2, with adifferent filtering capability disclosed. Like referenced elementsoperate similar to those described with respect to FIG. 2.

As illustrated in FIG. 22, the filtering pixel 2200 includes asample-and-hold filter 2210. The sample-and-hold filter 2210 enablespseudo-digital sampling in the analog domain. The output from the bufferFET 204 (or source follower) goes through the sample-and-hold circuit2210, which stores a sample of a previous intensity of light falling onthe photodiode 201. A sample FET 2201 is included, which can becontrolled to identify when a sample of the output from the buffer FET204 is to be taken and stored in a sample capacitor 2202. The length ofthe sample can be controlled by controlling the operational voltageapplied to the sample FET 2201. In some embodiments, the sample may be aone-time, instantaneous sample of the output from the buffer FET 204,while in other embodiments the sample FET 2201 may be held open for aduration to accumulate a longer sample. The sampling rate depends on theparticular implementation and the information of the signal desired tobe identified and processed. The sample-and-hold circuit 2210 enables awide range of sampling-based filtering to be applied in the analogdomain. A sample source follower 2204 serves to isolate the samplecapacitor 2202 from the rest of the circuit.

The current mirror 2220 serves to copy a current from one active devicein a circuit to another active device, ensuring that the output currentof the mirror remains constant regardless of loading on the circuit. Insome embodiments, the resistor 2203 may be implemented as a switchedcapacitor. The resistor 2203 is used to compare the sampled voltageoutput from the FET 2204 with the instantaneous voltage from the FET 204to set a current input to the current mirror 2220. The voltage from theresistor 2203 serves to control the current mirror FETs. The sample fromthe sample FET 2201 is outputted to the current mirror 2220 through anoutput FET 2204. The current mirror 2220 compares the voltages of thesample from the sample capacitor 2203 with the voltage of the outputfrom the buffer FET 204 to generate a magnitude of the current. That is,the current mirror 2220 generates a current (through resistor 2203) witha magnitude proportional to the different between the voltage of theoutput from the buffer FET 204 and the output from the sample buffer FET2204 (i.e., the sample held in the sample capacitor 2202).

As the current mirror 2220 generates a current proportional to thedifference in voltage between the sample and the output from thephotodiode 201, it represents the average intensity of light between thetwo times. In other words, the current at the current mirror 2220 isproportional to the current on the photodiode 201. If no light falls onthe photodiode 201, the sample and the output from the photodiode 201are equal, and no current flows through the current mirror 2220. Bycreating a copy of the current generated by the photodiode, it frees usup to operate on this current using pseudo-digital analog filtering.Using current subtraction allows for the current to be added orsubtracted, either by charging or discharging a capacitor 2231 of acurrent subtractor 2230. Two operation FETs 2232 and 2233 are used tobias one side or the other of capacitor 2231, are used in conjunctionwith two FETs 2234 and 2235 that open up the current output to one sideor the other of capacitor 2231 from the current mirror 2220. That is,the voltage across the capacitor 2231 increases when FET 2232 and FET2234 are selected (charging), while the voltage across the capacitor2231 decreases when FET 2233 and FET 2235 are selected (discharging). Avoltage 2240 (Vc) is used to control the FETs 2232, 2234, and voltage2250 (Vs) is used to control the FETs 2233, 2235.

One skilled in the art will realize that many other circuit designs fortaking out the current from the photodiode 201 exist. For example, thecurrent from the photodiode can be taken out directly and fed into thecurrent subtractor 2230 to perform the filtering.

The coefficients for the filter are set by determining the sign andmagnitude. The “sign” refers to the polarization of the filter. The signof the coefficient is selected by the digital signals that control theFETs 2232, 2233, 2234 and 2235. If the capacitor 2231 is charged, thecoefficient is positive, and if the capacitor 2231 is discharged, thecoefficient is negative. In various embodiments, the sign of the filtermay be synchronized with a modulation scheme, such that a change in thesign of the filter is tied to a specific pattern of a modulated source.In other embodiments, the sign of the filter may be determined in anasynchronous manner, allowing the filter to be programmed independentlyof a particular modulation scheme utilized. The magnitude of thecoefficient is set by adjusting the resistor 2203, which, if using aswitched capacitor, can be changed by adjusting the switching frequency.The magnitude may be adjusted in the same way as the sign discussedabove (i.e., synchronously or asynchronously).

The current subtractor 2230 is similar to the differentiator 205discussed with respect to FIG. 2. A differentiator is a circuit that isdesigned such that the output is proportional to the rate of change. Inthe example discussed above with respect to FIG. 2, the resistor andcapacitor served to output the derivative of the intensity of the lightfalling on the photodiode directly (i.e., on the continuous signal). Inthe example filtering pixel of FIG. 22, the current subtractor 2230outputs the proportional rate of change in the sampled signal when onecurrent sample is charging the capacitor 2231 and the next isdischarging the capacitor 2231. In various embodiments, the currentsubtractor 2230 of FIG. 22 may be replaced with a differentiator 205 tooutput the derivative of the sampled signal. In some embodiments, thedifferentiator 205 in FIG. 2 can be replaced with a current subtractor2230 as discussed with respect to FIG. 22 to provide the intensity ofthe light minus any unwanted intensity.

Each filtering pixel 2200 includes its own integrator, comprising FET212 and capacitor 213. The integrator serves to produce a representativeoutput of the filtering output from the subtractor 2230. By including anintegrator with each filtering pixel 2200 enables the specific signalfrom each filter to be read out over the read bus 216, as opposed tosimply outputting the signal falling on the photodiode as in atraditional pixel array. Based on operation of read out FET 214 usingthe select bus 215, the filtered signal can be outputted on the read bus216.

Through operation of the FETs 2232, 2233, 2234, 2235, the filteringpixel 2200 may be programmable, enabling different types of outputsignals to be read out on the read bus 216. For example, in variousembodiments the filtering pixel 2200 may be programmed to function as atraditional pixel (i.e., simply outputting the total intensity fallingon the photodiode over a period of time) by setting the FETs 2232, 2233,2234, 2235 such that the capacitor 2231 is always charging, and thenoutputting the total intensity through the FET 212. In otherembodiments, the FETS 2232, 2233, 2234, 2235 may be programmed asdiscussed above to add or subtract samples.

FIGS. 23A and 23B illustrate example implementations of the filteringpixel described with respect to FIG. 22. The diagrams in FIGS. 23A and23B are similar to the diagram of the general operation illustrated inFIG. 21, with the sample-and-hold 2320 and subtractor 2330 comprisingthe filter 2130 discussed with respect to FIG. 21. The diagram of FIG.23A illustrates the filtering pixel 2200 as illustrated in FIG. 22. Thatis, the output from the photodiode 2310 goes through the sample-and-holdfilter 2320 which takes a sample of the output from the photodiode 2310.The sample is compared against the instantaneous voltage from thephotodiode 2310 in the subtractor 2330. In various embodiments, thecurrent mirror discussed with respect to FIG. 22 may be included in thesample and hold 2320 or the subtractor 2330. The output from thesubtractor 2330 is then integrated in the integrator 2340.

In FIG. 23B, a variation is shown wherein the difference between varioussamples is filtered. That is, the sample and hold 2320 includes two ormore sample and hold filters like the sample and hold filter 2210discussed with respect to FIG. 22.

FIG. 28 illustrates an example embodiment of a filtering pixel 2800 withmultiple sample and hold filters in accordance with various embodimentsof the technology disclosed herein. As illustrated in FIG. 28, thefiltering pixel 2800 is similar to the filtering pixel 2200 discussedwith respect to FIG. 22, and similar referenced elements function in asimilar manner. The filtering pixel 2800 includes a second sample andhold filter 2810, comprising a second sample FET 2801, sample capacitor2802, and a sample source follower FET 2804. In this way, the differencebetween different samples (i.e., the difference between the sample fromsample and hold filter 2210 and the sample from sample and hold filter2810) can be filtered and used to set a current input to the currentmirror 2220.

The number of sample and hold filters that can be implemented within thecircuit is not limited to only two. In various embodiments, thefiltering pixel 2800 may include more than two sample and hold filters,with operational switches for determining which samples will be comparedfor determining the current to be outputted by the current mirror 2220.A person of ordinary skill in the art in view of the current disclosurewould understand how to connect multiple sample and hold filters in apixel.

To filter out unwanted light sources, one method involves the use of amodulated light source. FIG. 24 illustrates an example imaging system2400 implementing an image sensor with filtering pixels configured withsuch filtering capability. The imaging system 2400 includes a lightsource 2410, an image sensor 2420, and a lens 2430. In the illustratedembodiment, the light source 2410 is a modulated light source. In thismanner, light with a known modulation scheme 2440 can outputted into thescene sought to be captured by the image sensor 2420. Because themodulation 2440 is known, the filtering pixels of the image sensor 2420may be configured to use the known modulated light to identify unwantedlight sources within the captured light and remove those. In variousembodiments, the light outputted from the light source 2410 may beamplitude modulated, phase modulated, or polarization modulated. Invarious embodiments, the modulation 2440 may be a square wave, pulsetrain, sinusodial wave, or pseudo-random. In various embodiments, themodulation 2440 is not periodic over the entire frame. In variousembodiments, the modulation 2440 may be coded, similar to code-divisionmultiplexing used in cellular communication. In this way, the filteringpixel of the image sensor 2420 may be programmable, and may selectdifferent codes to filter out different unmodulated light. The type ofmodulation may depend on the type of unwanted light sought to befiltered out by the filtering pixels of the image sensor 2420.

The image sensor 2420 may include an array comprising filtering pixelscapable of filtering out undesired light sources. In variousembodiments, the array may consist of filtering pixels similar to thefiltering pixel 2200 discussed with respect to FIG. 22. In otherembodiments, the array may include additional filtering pixels withother filtering capabilities, such as any of the example filteringpixels discussed with respect to FIGS. 2-13.

In various embodiments, the same modulation scheme 2440 used to modulatethe light source 2410 may be used to drive the sample-and-hold andcurrent subtraction portions of the filtering pixel 2200 embodied in theimage sensor 2420. In this way, the filtering pixel is capable ofsampling the signal from the photodiode in conjunction with themodulation. The modulation allows the filtering pixel to key in on theaspects of the captured light that is desired. In the illustratedembodiment, the light source 2410 and the filtering pixels of the imagesensor 2420 are synchronized. In other embodiments, the light source2410 and the filtering pixels of the image sensor 2420 need not besynchronized, such as when different coded schemes may be used with themodulation 2440. In such embodiments, the filtering pixels may beconfigured to be able to detect a particular coding scheme being usedand drive the filtering components accordingly.

FIG. 25A illustrates when the filtering pixel is configured to determinewhen an object is illuminated and remove unwanted light sources. In theillustrated example, the modulated illumination is aligned such that thepolarity of the filter is aligned with the ON state of the modulation.Any non-modulated light (or light at a different modulation) does notbuild up on the pixel as the filter rejects the background light (thatremains essentially constant to the pixel as it includes equal “−”and“+”). Only the modulated light source builds up as it is aligned withthe filter highs.

In addition to using the modulated light source for removing unwantedlight sources, this arrangement can be used to also determine when anobject is moving in the scene. FIG. 25B illustrates when the filteringpixel is configured to determine when an object is moving. As discussedabove, a derivative filter (like that discussed with respect to FIGS.1-13) could be included in the filtering pixel 2200 discussed above withrespect to FIG. 22. By using a modulated light source, the light can besampled such that only the modulated light is accumulated (filtering outunwanted lights sources). In addition, by flipping the polarity of thefilter in the middle of a frame, the derivative can be filtered out toshow motion. As shown in FIG. 25B, the polarity flips midway through theframe, such that the second half of the frame has the filter highs (“+”)aligned with the OFF state of the modulated light and the filter lows(“−”) are aligned with the ON state of the modulated light. Therefore,if the magnitude of the modulated light is not changing during theframe, it would be cancelled out similarly to unwanted light sources.However, if there is motion in the frame the change would be captured.In some embodiments, the polarity of the filter is changed more thanonce throughout a frame. In some embodiments, the polarity of the filteris not changed, but the illumination shifts midway through the frame,such that the second half of the frame has the filter highs (“+”) arealigned with the OFF state of the modulated light and the filter lows(“−”) are aligned with the ON state of the modulated light. In theseembodiments, the light modulation and the filter polarity work togetherto determine the filter characteristics. Similarly, to vary the filtercoefficient magnitude, either the filter amplitude can be modified foreach sample, or the light intensity or both.

Although FIGS. 25A and 25B illustrate two types of methods of switchingthe filter polarity compared to the modulated illumination, nothing inthis disclosure should be interpreted as limiting the technology to onlythose two arrangements. FIGS. 25C and 25D illustrate two additionalarrangements that are possible in accordance with the technologydisclosed herein. A person of ordinary skill in the art would understandthat the technology disclosed herein would allow for any number ofdifferent arrangements to be programmed into the filtering pixel,allowing for a wide range of outputs to be generated.

In various embodiments, the imaging system 2400 described with respectto FIG. 24 and the motion detection camera system 1400 discussed withrespect to FIG. 14 may be combined into a single imaging system. FIG. 26illustrates an example imaging system 2600 in accordance with variousembodiments of the technology disclosed herein. The imaging system 2600includes the components discussed with respect to FIG. 14, as well as anadditional light source 2610. The light source 2610 may be similar tothe light source 2410 discussed with respect to FIG. 24. In this way,the imaging system 2600 is configured both to enable out-of-plane motiondetection as discussed with respect to FIG. 14, as well as filtering outunwanted light sources as discussed with respect to FIG. 26. In variousembodiments, a single light source may be used to accomplish both theout-of-plane motion detection using Doppler shift as well as filteringout unwanted signals using a modulated light source.

Utilizing embodiments of the filtering pixels in accordance with thepresent disclosure, mission-specific imaging systems may be developedthat enable pseudo-digital processing in the analog domain. However,imaging systems capable of performing various different types offiltering may be created implementing in accordance with the technologydisclosed herein. For example, in various embodiments the pixel arrayimplemented in an image sensor may include multiple filtering pixels inaccordance with the filtering pixels discussed herein, where eachfiltering pixel is configured to perform a specific type of filtering.For example, the array may include filtering pixels including aderivative filter, filtering pixels including a double derivativefilter, and filtering pixels including a sample-and-hold/currentsubtraction filter. In some embodiments, the array may further include atraditional pixel. In various embodiments, a filtering pixel may beincluded that has more than one filter type included (e.g., includesboth a derivative filter and a sample-and-hold/current subtractionfilter).

FIG. 27 illustrates an example pixel array 2700 in accordance withembodiments of the technology disclosed herein. The example pixel array2700 may be implemented in a camera to provide filtering capabilities inthe pixel domain. As described above, an image sensor may be createdhaving a pixel array 2700 having multiple different types of pixels(e.g., traditional, filtering pixels with a derivative filter, filteringpixels with a sample-and-hold/current subtraction filter, etc.). Thevarious pixels may be arranged in a similar fashion as used for anoptical color filter, such as a Bayer filter. A Bayer filter is a colorfilter, where a filter is laid over the array of pixels such differentwavelengths of light corresponding to red, blue, and green light arecaptured by individual pixels. In this way, a single traditional pixelarray is capable of providing RBG images. In FIG. 27, the filteringpixel array 2700 includes a variety of different pixels. In variousembodiments, pixel 1 may be a traditional pixel, pixel 2 may be afiltering pixel with a single derivative filter, pixel 3 may be afiltering pixel with a double derivative filter, and pixel 4 may be afiltering pixel with a sample-and-hold/current subtraction filter. Inother embodiments, the arrangement of the pixels may differ. A person ofordinary skill in the art would understand that the pixels within thearray could be arranged in a multitude of different configurations, andthat more than 4 different types of pixels could be included. Nothing inthis disclosure should be interpreted to limit the scope only to arrayshaving 4 different types of pixels arranged as illustrated. In variousembodiments, each of the pixels in the array could be programmable, suchthat each pixel could be changed to perform a different type offiltering. For example, the pixel circuit of FIG. 21 has a programmablefilter, meaning the filter can be changed by altering the digitalsignals that select when the various FETs are open or closed.

EXAMPLE APPLICATIONS

As previously noted, the disclosed filtering pixel technology may beimplemented in a variety of imaging applications such as selected FOVcameras, three-dimensional cameras, four-dimensional structured lightcameras, four-dimensional laser scanning cameras, spectroscopy cameras,ambient light insensitive cameras, event based cameras, time of flightcameras, or some combination thereof. The filtering pixel technology maybe implemented in image sensors that are used in conjunction withcoherent or incoherent light sources. For example, the technologydisclosed herein may be implemented in conjunction with light emittingdiodes (LEDs), lasers, or other appropriate light sources.

Although applications for filtering images will primarily be describedin the context of using the filtering pixel technology disclosed herein,in alternative implementations, the above-mentioned imaging applicationsmay be implemented by applying digital filtering operations to thedigital output of a conventional image sensor (e.g., a CMOS sensor),where each pixel of the image sensor is the same. For example, digitalfiltering may be applied after one or more ADCs of the image sensor. Insuch implementations, digital filtering may be implemented using a fieldprogrammable gate array (FPGA), an application-specific integratedcircuit (ASIC), the logic of the image sensor itself (after the ADCoutput), a digital signal processor (DSP), a programmable logic circuit(PLC), a graphical processing unit (GPU), a central processing unit(CPU), or some other digital processor that may be configured to processthe digital output of an image sensor (e.g., multiple frames captured bythe sensor).

In digital filtering implementations, the output image may be composedof a series of captured digital image frames to which suitablealgorithms (e.g., image addition, subtraction, derivative, doublederivative, etc.) may be applied in the digital domain to identifyspecific components of the captured light. The image sensor may beconfigured to capture images at a sufficiently high frame rate (e.g.,greater than 1,000 frames/s, greater than 10,000 frame/s, or evengreater than 100,000 frame/s) such that the output image that iscomposed may suitably identify the desired components of the capturedlight and avoid artifacts introduced by moving objects. For example, theframe rate may meet or exceed the Nyquist sampling rate for capturingdesired information contained in the light collected by the imagesensor.

Selected FOV Imaging System

In embodiments, the filtering pixel technology described herein may beimplemented in a selected FOV imaging system. FIG. 29A illustrates onesuch example of a selected FOV imaging system 2900 implementing an imagesensor 2920 with filtering pixels. Imaging system 2900 includes a lightsource 2910, an image sensor 2920, a lens 2930, and a modulationcomponent 2940. In the example implementation of FIG. 29A, imagingsystem 2910 may be configured such that it only “sees” what is beingilluminated by light source 2910, even in the presence of external lightsources, including bright day light (e.g., sunlight), other outdoorlight sources, and/or indoor light sources. To this end, the opticalbeam output by light source 2910 may be modulated using a particularmodulation code or scheme, and the filtering pixels of image sensor 2920may be configured such that image sensor 2920 only detects light that ismodulated with the same code. For example, the filtering pixels of imagesensor 2920 may configured to integrate signals generated from anincident light source having the code. Any signals generated by lightincident on a photodetector of a filtering pixel that does not have thecode may be filtered out before integration. In various implementations,the modulation of the light source and the pixel filters may besynchronized such that the filtering pixels are configured to detectlight having the modulation code of the light source.

FIG. 29B is a block diagram illustrating an example modulation component2940 that may be used to modulate light source 2910 and synchronizemodulation of the light source with configuration of the filteringpixels of the image sensor 2920. As shown, modulation component 2940 mayinclude a controller 2941, a light source driver 2942, and a modulationcode storage 2943 that may store one or more modulation codes to applyto an optical beam. In various implementations, the stored modulationcode(s) may correspond to an information signal that may cause aparticular amplitude modulation, phase modulation, frequency modulation,and/or polarization modulation of an optical beam. The modulation codemay be associated with a square waveform, a pulsed waveform, asinusoidal waveform, a pseudo-random waveform, or some other waveform.The waveform may be periodic over the length of an image frame of imagesensor 2920 or it may not be periodic. In some implementations, themodulation code may be coded similar to codes used in code-divisionmultiplexing used in cellular communication.

During operation, controller 2941 may send a signal to light sourcedriver 2942, including the modulation code, to be modulated on anoptical beam transmitted by light source 2910. In some instances, themodulation code retrieved from modulation code storage 2943 may beconverted to an appropriate format for optical transmission prior tocontroller 2941 signaling driver 2942. Light-source driver 2942 mayaccept the modulation code and output an appropriate modulatedelectrical signal to drive the light source 2910 to output an opticalbeam modulated with the modulation code, using power supplied from apower supply (not shown).

Controller may also send a synchronization signal to the filteringpixels of image sensor 2920 such that the pixels are configured todetect only light modulated with the modulation code. For example, inimplementations where the filtering pixels are programmable (e.g., asdescribed above with reference to filtering pixels 2100, 2200, and2800), the control synchronization signal may configure the filteringpixels to detect only light modulated with the modulation code byconfiguring coefficients of the filter as described above. In someembodiments, the controller may receive a synchronization signal fromthe image sensor.

FIG. 30A is an operational flow diagram illustrating an example methodthat may be implemented by imaging system 2900 to provide selected FOVimaging. At operation 3010, an electrical signal is outputted (e.g., bydriver 2942) to drive light source 2910 based on a modulation code. Atoperation 3020, a modulated optical beam carrying the code is generatedby light source 2910 based on the electrical drive signal. At operation3030, a plurality of filtering pixels of the image sensor 2920 may beconfigured to detect only light modulated with the modulation code. Forexample, the filters of the filtering pixels may be programmed to filterout signals from incident light that does not carry the modulation code.In some implementations, a controller 2941 of a modulation component2940 may cause the filtering pixels to be programmed in synchronizationwith modulation of the light source.

At operation 3040, light is received at the image sensor 2920. Thereceived light may include, for example, modulated light reflected froman object illuminated with the modulated light source, and otherexternal sources of light such as natural sunlight that are notmodulated with the modulation code or light sources modulated withdifferent codes. At operation 3050, filtering pixels of image sensor2920 may detect light modulated with the modulation code and filter outother sources of light that are not modulated with the modulation code.For example, for a given filtering pixel, light incident on aphotodetector 2120 may be filtered by a filter 2130 such that only lightmodulated with the modulation code is integrated by integrator 2140.Following collection of light at the image sensor 2920, at operation3060, an image may be generated of only the scene and/or object that wasilluminated by the modulated light source.

In alternative implementations, the method of FIG. 30A may be performedby applying digital filtering to the digital output of an image sensorwithout filtering pixels (e.g., a CMOS image sensor) that has asufficiently high frame rate. In such implementations, operations 3030and 3050 may instead be performed by configuring a digital filter tofilter out the digital output of the image sensor that is not modulatedwith the code.

FIG. 30B illustrates a representation of an example selected FOV imagethat may be captured in accordance with implementations. FIG. 30Cillustrates a representation of the corresponding image captured with astandard camera. As illustrated, by detecting only light modulated witha specific code, only the person's hand may be illuminated and detectedwhereas ambient light changes may filtered out and ignored.

FIG. 31A illustrates another example of a selected FOV imaging system3100 implementing an image sensor 3120 with filtering pixels. Imagingsystem 3100 includes light sources 3111 and 3112, an image sensor 3120,a lens 3130, and a modulation component 3140. In the exampleimplementation of FIG. 31A, imaging system 3100 may be configured suchthat it only “sees” what is being illuminated by both light sources 3111and 3112, further limiting the field of view (in depth as well). To thisend, the optical beam output by light source 3111 may be modulated witha first modulation code and the optical beam output by light source 3112may be modulated with a second modulation code. Some of the filteringpixels of image sensor 3120 may be configured such that they only detectlight that is modulated with the first code, some of the filteringpixels of image sensor 3120 may be configured such that they only detectlight that is modulated with the second code, some filtering pixels ofthe image sensor 3120 may be configured such that they only detect lightthat is not modulated (e.g., to detect background illumination only),some filtering pixels of the image sensor 3120 may be configured suchthat they detect light modulated with either code (e.g., to detectportions of the scene illuminated by either light source), and somefiltering pixels of the image sensor 3120 may be configured such thatthey detect light modulated with both codes at the same time (e.g., todetect only portions of the scene that reflect light from both lightsources). In various implementations, the modulation of the light sourceand the pixel filters may be synchronized such that the filtering pixelsare configured to detect light having the first or second modulationcodes.

In some implementations, light sources 3111 and 3112 may emit light indifferent wavelengths (e.g., colors), polarizations, etc. such that theimage sensor pixels can distinguish the light source using an opticalfilter, in addition to the time based filter. In some implementations,there may be more than two light sources (e.g., three, four, or morelight sources) that are modulated with either the same modulationcode(s) or different modulation code(s).

FIG. 31B is a block diagram illustrating an example modulation component3140 that may be used to modulate light sources 3111 and 3112 andsynchronize modulation of the light sources with configuration of thefiltering pixels of the image sensor 3120. As shown, modulationcomponent 3140 may include a controller 3141, a light source driver3142, a light source driver 3143, and a modulation code storage 3144that may store modulation codes to apply to each of the optical beamsgenerated by light sources 3111 and 3112.

During operation, controller 3141 may send a signal to each of lightsource drivers 3142 and 3143, including respective modulation codes, tobe modulated on respective optical beams transmitted by each of lightsources 3111 and 3112. Each of light source drivers 3142 and 3143 mayaccept the respective modulation codes and output an appropriatemodulated electrical signal to drive the respective light source 3111 or3112 to output an optical beam modulated with the respective modulationcode. Controller may also send a first synchronization signal to a firstset of filtering pixels of image sensor 3120 such that first set offiltering pixels are configured to detect only light modulated with thefirst modulation code, and a second synchronization signal to a secondset of filtering pixels of image sensor 3120 such that the second set offiltering pixels are configured to detect only light modulated with thesecond modulation code.

FIG. 32 is an operational flow diagram illustrating an example methodthat may be implemented by imaging system 3100 to provide selected FOVimaging using two modulated light sources. At operation 3210, modulatedelectrical signals are outputted (e.g., by drivers 3142 and 3143) todrive light sources 3111 and 3112 based on respective first and secondmodulation codes. At operation 3220, first and second modulated opticalbeams respectively carrying the first and second modulation codes may begenerated by a respective one of the light sources.

At operation 3230, filtering pixels of the image sensor 3110 may beconfigured to detect only light modulated with the first modulationcode, only light modulated with the second modulation code, lightmodulated with either the first or second code, light modulated onlywith both the first and second codes at the same time, and/or light notmodulated with either the first or second codes. For example, a firstplurality of filtering pixels may be programmed to filter out signalsfrom incident light that does not carry the first modulation code, asecond plurality of the filtering pixels may be programmed to filter outsignals from incident light that does not carry the second modulationcode, and a third plurality of filtering pixels may be programmed tofilter out signals that carry the first or second codes (e.g., to detectbackground illumination). In this example, the three sets of filteringpixels may be configured to be arranged in an alternating pattern on theimage sensor array, illustrated by FIG. 31C. In another example, theimage sensor may be configured with only two sets of filtering pixels(e.g., a first set that detects only light modulated with the firstcode, and a second set that detects only light modulated with the secondcode). As would be appreciated from the foregoing description, up tofive different sets of filtering pixels may be programmed in thisexample implementation.

At operation 3240, light is received at the image sensor 3120. Thereceived light may include, for example, modulated light reflected froman object illuminated with the first modulated light source and/or thesecond modulated light source, and other external sources of light thatare not modulated with the first or second modulation codes. Atoperation 3250, the configured filtering pixels detect light modulatedwith only the first code, light modulated only with the second code,light modulated with either the first or second code, light modulatedwith the both the first and second codes at the same time, and/or lightnot modulated with the first or second codes. For example, a firstplurality of filtering pixels may detect light modulated with the firstcode and a second plurality of filtering pixels may detect lightmodulated with second code. As such, in this example, only objectsilluminated by both the first and second modulated light sources may beseen by all pixels of the image sensor 3110. Following collection oflight at the image sensor 3110, at operation 3260, an image may begenerated of the scene and/or object that was illuminated by bothmodulated light sources.

In alternative implementations, the method of FIG. 32 may be performedby applying digital filtering to the digital output of an image sensorwithout filtering pixels (e.g., a CMOS image sensor) that has asufficiently high frame rate. In such implementations, operations 3230and 3250 may instead be performed by configuring one or more digitalfilters to filter out the digital output of the image sensor that is notmodulated with the first code, to filter out the digital output that isnot modulated with the second code, to filter out the digital outputthat is not modulated with the first or second codes, to filter out thedigital output that is not modulated with both the first and secondcodes, and/or to filter out the digital output that is modulated witheither code.

Although examples described above with reference to the selective FOVimaging system have been described with reference to systems thatutilize one or two different modulated light sources, it should be notedthat any number of modulated light sources may be used with the samenumber of corresponding sets of filtering pixels of the image sensorthat are configured to detect a respective one or more of the modulatedlight sources. For example, for a system that uses n modulation codeswith n modulated light sources, the image sensor may be configured tohave up to 2^(n) sets of filtering pixels, where each of the sets offiltering pixels detects between 0 and n modulated light sources.

Three Dimensional Imaging System

In embodiments, the filtering pixel technology described herein may beimplemented in a three-dimensional (3D) imaging system. The 3D imagingsystem may be configured to create a two-dimensional spatial image ofobjects (e.g., image having x and y coordinates) and detect theevolution of those objects over time. To this end, an imaging systemhaving a configuration similar to that illustrated in FIGS. 29A-29B maybe implemented. In particular, the imaging system may include a lightsource, an image sensor, a lens, and a modulation component thatmodulates the light with a modulation code and synchronizes modulationwith configuration of the filtering pixels. Similar to the exampledescribed above with respect to imaging system 2900, the imaging systemmay be configured such that all of its filtering pixels detect onlylight that is modulated with the modulation code. As such, detection maybe focused on only objects illuminated by the modulated light source.

In addition, three different sets of filtering pixels of the imagesensor may be configured to perform one of three respective functions:output a signal proportional to the intensity of the received light(i.e., like a traditional image sensor), output a signal proportional toan increase in the intensity of the received light (e.g., motiondetection in one direction), and output a signal proportional to adecrease in the intensity of the received light (e.g., motion detectionin another direction). In some implementations, only one or two of thesesets of filtering pixels may be used.

FIG. 33A is an operational flow diagram illustrating an example methodthat may be implemented by a 3D imaging system in accordance withembodiments. Similar to operations 3010 through 3030 of FIG. 30, atoperations 3310 through 3330, the method may include: outputting amodulated electrical signal to drive the light source based on amodulation code; generating a modulated optical beam at the light sourcebased on the modulation code; and configuring the filtering pixels ofthe image sensor to detect only light modulated with the code. In theexample of FIG. 35, all pixels may be configured to detect only lightmodulated with the code such that only illuminated objects of interestare imaged in two spatial dimensions and their evolution is detectedover time.

At operation 3340, a first set of filtering pixels may be configured tooutput a signal proportional to the intensity of received light. Theoutput from this set of filtering pixels may be used to generate atwo-dimensional image of an illuminated object of interest. At operation3350, a second set of filtering pixels may be configured to output asignal proportional to an increase in the intensity of the receivedlight. The output from this set of filtering pixels may be used todetect motion toward the pixel. At operation 3360, a third set offiltering pixels may be configured to output a signal proportional to adecrease in the intensity of the received light. The output from thisset of filtering pixels may be used to detect motion away from thepixel. As discussed above, the three sets of filtering pixels may beprogrammed in accordance with the techniques described herein. Forexample, the first set of filtering pixels may be programmed tointegrate a light signal that carries the modulation code, the secondset of filtering pixels may be programmed to apply a positive derivativefilter to a light signal that carries the modulation code, and thesecond set of filtering pixels may be programmed to apply a negativederivative filter to a light signal that carries the modulation code.

At operation 3370, light may be received at the image sensor (e.g.modulated light reflected by illuminated object and other externalsources of light), and a 3D image of the object illuminated with themodulated light may be created using the filtering pixel configurations.

In alternative implementations, the method of FIG. 33A may be performedby applying digital filtering to the digital output of an image sensorwithout filtering pixels (e.g., a CMOS image sensor) that has asufficiently high frame rate. In such implementations, operations3330-3360 may instead be performed by configuring one or more digitalfilters to filter out the digital output of the image sensor. Forexample, a decrease or increase in signal intensity may be detected byadding or subtracting consecutive digital image frames.

By way of example, FIG. 33B illustrates a representation of a 3D imagethat may be captured for motion sensing in accordance with oneimplementation. FIG. 33C illustrates a representation of thecorresponding image captured with a standard camera. As illustrated, byconfiguring filtering pixels in accordance with implementations, motionsthat are missed by a standard camera may be detected. In this example,the trajectory of a person's hand is detected, where white correspondsto the hand's start position and black corresponds to the hand's endposition. The example image illustrated by FIG. 33B may be captured byconfiguring the filtering pixels with a filter that consecutively addsand subtracts signals to output a signal proportional to both positive(white) and negative (black) changes in the received light such that thepath that the hand follows may be determined by seeing many black andwhite snapshots of the hand all together in a single image.

Four Dimensional Imaging System

In embodiments, the filtering pixel technology described herein may beimplemented in a four-dimensional (4D) imaging system. The 4D imagingsystem may be configured to create a three-dimensional spatial image ofobjects (e.g., image having x, y, z coordinates) and detect theevolution of those objects over time.

In one implementation, the 4D imaging system may be implemented as astructured light imaging system. FIG. 34 illustrates one such exampleimplementation of a structured light imaging system 3400. Asillustrated, imaging system 3400 may include a light source 3410, animage sensor 3420, an optical lens 3430, a modulation component 3440,and a grating 3450. During operation, structured light imaging system3400 may generate a pattern of modulated light using grating 3450 orother suitable device for generating a pattern of light. By determininga displacement of the patterns of light, the 3D coordinates of thedetails of the object's surface may be subsequently determined.

FIG. 35 is an operational flow diagram illustrating an example methodthat may be implemented by a 4D structured light imaging system 3400 inaccordance with embodiments. For simplicity of discussion, repeated oranalogous operations of the method of FIG. 34 are listed withcorresponding reference numbers. As illustrated, in addition to theexample operations described above with reference to the method of FIG.34, in the method of FIG. 35 imaging system 3400 is configured to passthe modulated optical beam through a pattern generator at operation3510. For example, the pattern generator may be a grating 3450 or otherdevice for generating a pattern of light (e.g., a stripe pattern orother suitable pattern) that may be used to create a 3D spatial image ofthe object. Additionally, at operation 3520, based on the receivedpattern of light, the image sensor may generate a 4D image of the objectilluminated with the modulated light. For example, when a 3D object ismoving in front of the camera, the vertical stripes that are projectedon the object shift as the object moves. The stripes may move to theleft (from the camera point of view) when the object moves closer to thecamera. Conversely, the stripes may move to the right when the objectmoves further away from the camera. As the object moves laterally, newstripes may appear in the space to which the object moves. Conversely,stripes may disappear in the space the moving object leaves behind. Allof these changes may be detected by the filtering pixels that senseincreases and/or decreases in intensity during the frame. Byinterpreting both the shape of the stripes and how these stripes move, a4D rendering of the object can be produced.

In alternative implementations, the method of FIG. 35 may be performedby applying digital filtering to the digital output of an image sensorwithout filtering pixels (e.g., a CMOS image sensor) that has asufficiently high frame rate. In such implementations, operations3330-3360 may instead be performed by configuring one or more digitalfilters to filter out the digital output of the image sensor. Forexample, a decrease or increase in signal intensity may be detected byadding or subtracting consecutive digital image frames.

In some implementations, the 4D imaging system may be implemented as alaser scanning system. FIG. 36 illustrates one such exampleimplementation of a 4D laser scanning imaging system 3600. The scanningsystem may include a light source 3610, a camera 3620 include areceiving lens and image sensor with filtering pixels, and modulationcomponent 3630. The light source 3610 of the 4D imaging system mayimplemented as a laser line scanner. For example, the laser line scannermay be a single line or stripe scanner or a multi-stripe scanner.

During operation, light source 3610 may output a stripe-shaped opticalbeam that is scanned along the surface of an object. For example, theobject may be scanned at a constant frequency. The image sensor ofcamera 3620 may be configured with four different sets of filteringpixels. The filters of the first set of filtering pixels may configuredsuch that they output a peak signal when a phase of light incident onthe pixel (e.g., light incident on the pixel's photodetector) is zerodegrees. The filters of the second set of filtering pixels mayconfigured such that they output a peak signal when a phase of lightincident on the pixel is 90 degrees. The filters of the third set offiltering pixels may configured such that they output a peak signal whena phase of light incident on the pixel is 180 degrees. The filters ofthe fourth set of filtering pixels may configured such that they outputa peak signal when a phase of light incident on the pixel is 270degrees.

By configuring the different sets of filtering pixels in this manner,and since the phase that each pixel sees is a function of the lookingangle of the pixel and the angle of the scanning light source when itilluminates the portion of the object the pixel is imaging, the threedimensional profile of a scanned surface of an object may be determinedusing triangulation. For a particular pixel location on camera 3620, thedistance to the scanned object may be determined from the phase of thelight incident on the pixel. The phase of the incident line may bedetermined by comparing the intensity of the four filtering pixels thatoutput a peak signal at different phases of incident light.

FIG. 37 illustrates the pixel filter of the four filtering pixelsconfigured in this manner (shown in black) as compared to the lightpeaks reflected from the object at the location the pixel is looking at(shown in light gray). For each of the four plots, the horizontal axisrepresents time and the vertical axis represents intensity (for thelight peaks) and filter coefficient (0 for block or 1 for pass). In thisexample, each light peak corresponds to when the light stripe (e.g., thelight stripe illustrated by FIG. 36) passes through the portion of theobject the four pixels are looking at. Since the four pixels are assumedto be close together, the light passes illuminates each of the fourpixels at the same time. However, because each of the four pixel filtershas a different phase, the integrated signal from each pixel isdifferent.

In some embodiments, illustrated by FIG. 37, the pixel frequency may beslightly offset from the illumination frequency to create a beatfrequency such that pixel output varies sinusoidally with phase. Asillustrated in this example, for Pixel 1, the light pulse starts insidethe filter's pass, so the entire pulse is recorded. Halfway through theintegration, the illumination pulse is halfway inside the pass andhalfway in the block portions of the filter, so only half of the pulseintensity is integrated. By the end of integration, the illuminationpulse is entirely in the block portion of the filter. As a result, theoutput of Pixel 1 will be roughly half of what it would be if it wasintegrating continuously without any filter. For Pixel 2, the lightpulse is always within the pass portion of the filter, so the integratedsignal will be the same as if it was integrating continuously. For Pixel3, the light pulse starts in the block portion of the filter and ends upin the pass portion of the filter, so the integrated signal will be thesame as Pixel 1. For Pixel 4, the light pulse remains in the blockportion of the filter during the entire frame, so the output of Pixel 4will be nearly zero. The phase at the center of the frame is 180degrees, indicating that this example set of four pixels are looking atthe center of the laser line scan, from which the distance may becomputed by triangulation, using the position of the laser line source,the position of the camera, and the looking angle corresponding to thelocation of the set of four pixels on the image sensor plane.

Spectroscopy Imaging System and Color Based Imaging

In embodiments, the filtering pixel technology described herein may beimplemented in a spectroscopy imaging system. FIG. 38 illustrates onesuch example of a spectroscopy imaging system 3900. As illustrated,spectroscopy imaging system 3900 may include a light source 3910configured to output multiple wavebands of light intensity modulated asa function of time with different codes, an image sensor 3920, anoptical lens 3930, and a modulation component 3940.

During operation of spectroscopy imaging system 3900, modulationcomponent 3940 may modulate different wavebands (e.g., visible colorssuch as red, green, blue, non-visible colors such as NIR, etc.) of lightoutput by light source 3910 by phase, frequency, using pulses, spreadspectrum techniques, or by some other technique. As illustrated in theexample of FIG. 38, the different wavebands of light are pulses withsame repetition frequency and different phase. For example, the firstwaveband may correspond to red light modulated with a first phase, thesecond waveband may correspond to green light modulated with a secondphase, and the third waveband may correspond to blue light modulatedwith a third phase. In implementations, the modulation codes for each ofthe transmitted wavebands may be orthogonal (e.g., phases may beseparated by 90 degrees) so that the pulses for each color do not appearat the same time.

Each filtering pixel in image sensor 3920 may be configured with afilter that detects only a waveband of light having a matching code.This is illustrated by FIGS. 39A-39C, which illustrate filtering basedon the phase modulation of each waveband. As shown in the example ofFIG. 39A, a filtering pixel 3921 is configured to filter current pulsesgenerated from light incident on pixel's photodetector 3922. In thisexample, the filter passes the current pulses corresponding to incidentlight from the third waveband of FIG. 38 and blocks the current pulsescorresponding to incident light from the other two wavebands. Thecapacitor 3923 stores energy from current pulses generated from incidentlight of the third waveband of FIG. 38. The voltage output and read outfrom filtering pixel 3921 is proportional to the integral of all currentpulses. The filter of pixel 3921 is configured to filter current pulsesbased on the modulation code (e.g., phase) of the third waveband oflight.

As shown in the example of FIG. 39B, filtering pixel 3921 is nowconfigured to filter current pulses based on the modulation code of thefirst waveband of light illustrated in FIG. 38. As such, capacitor 3923stores energy from current pulses generated from incident light of thethird waveband.

As the foregoing example illustrates, by implementing filtering pixelsin accordance with the technology disclosed herein, image sensor pixelsmay be configured with electronic decoding filters that detect aparticular color of light having a matching code. These electronicdecoding filters may be utilized in place of the traditional opticalcolor filters utilized in traditional image sensors. Such filteringpixels may be used in a wide variety of applications.

For example, in some implementations, such filtering pixels may be usedin surgical applications. For instance, in one surgical application,such pixels may be used to target specific spectral reflectancecharacteristics of a type of tissue (e.g., a tumor) to simplifydetection for precise and complete removal. In another surgicalapplication, such filtering pixels may permit surgeons to view 3D imagesin false color that represents tissue in a near IR or UV range. In otherexample surgical applications, such filtering pixels may be used tosense true color to provide a more accurate image to a surgeon (e.g., bydetecting red, green, and blue on the same pixel to provide precisewhite balance), to sense false color such that a surgeon can enhancecertain features, and/or to enhance color uniformity by using a tightlycontrolled spectrum. Other example surgical applications in which suchfiltering pixels may be utilized include periodic motion sensing (e.g.,to detect blood flow, blood pressure, heart beat, etc.), fast motionsensing (e.g., to detect bleeding or other fast changes as a surgeonoperates), tissue characterization (e.g., to detect tissue parameterssuch as stiffness by measuring motion response to force), and 3D sensingto map out the surface of tissue in 3D to aid a surgeon.

In yet further implementations, image sensor pixels designed withelectronic decoding filters that detect a particular color of lighthaving a matching code may be utilized in a variety of fluorescenceimaging applications. Luminescence of targets of interests that are dyedmay be detected by pixels based on a code of light excitation receivedat the image sensor.

It should be noted that although embodiments described herein have beenprimarily be described with reference to modulating the optical beamoutput by a light source of an imaging system, in some implementationsit may be possible to modulate the light received at an image sensor ofthe imaging system by using one or more optical filters (e.g.,polarizers, dichroic filters, monochromatic filters, infrared filters,bandpass filters, UV filters, infrared filters, etc.) that are placed inthe optical path before the image sensor. In such implementations, thefiltering pixels may be configured to detect only incident light havingmodulation properties consistent with these filters.

As used herein, the term component might describe a given unit offunctionality that can be performed in accordance with one or moreembodiments of the technology disclosed herein. As used herein, acomponent might be implemented utilizing any form of hardware, software,or a combination thereof. For example, one or more processors,controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logical components,software routines or other mechanisms might be implemented to make up acomponent. In implementation, the various components described hereinmight be implemented as discrete components or the functions andfeatures described can be shared in part or in total among one or morecomponents. In other words, as would be apparent to one of ordinaryskill in the art after reading this description, the various featuresand functionality described herein may be implemented in any givenapplication and can be implemented in one or more separate or sharedcomponents in various combinations and permutations. Even though variousfeatures or elements of functionality may be individually described orclaimed as separate components, one of ordinary skill in the art willunderstand that these features and functionality can be shared among oneor more common software and hardware elements, and such descriptionshall not require or imply that separate hardware or software componentsare used to implement such features or functionality.

As discussed above, some post-processing of the signal may be used insome embodiments to augment or provide additional information based onthe signal outputted from the filtering pixels. Where components orcomponents of the technology are implemented in whole or in part usingsoftware, in one embodiment, these software elements can be implementedto operate with a computing or processing component capable of carryingout the functionality described with respect thereto. One such examplecomputing component is shown in FIG. 19. Various embodiments aredescribed in terms of this example-computing component 1900. Afterreading this description, it will become apparent to a person skilled inthe relevant art how to implement the technology using other computingcomponents or architectures.

Referring now to FIG. 19, computing component 1900 may represent, forexample, computing or processing capabilities found within desktop,laptop and notebook computers; hand-held computing devices (PDA's, smartphones, cell phones, palmtops, etc.); mainframes, supercomputers,workstations or servers; or any other type of special-purpose orgeneral-purpose computing devices as may be desirable or appropriate fora given application or environment. Computing component 1900 might alsorepresent computing capabilities embedded within or otherwise availableto a given device. For example, a computing component might be found inother electronic devices such as, for example, digital cameras,navigation systems, cellular telephones, portable computing devices,modems, routers, WAPs, terminals and other electronic devices that mightinclude some form of processing capability.

Computing component 1900 might include, for example, one or moreprocessors, controllers, control components, or other processingdevices, such as a processor 1904. Processor 1904 might be implementedusing a general-purpose or special-purpose processing engine such as,for example, a microprocessor, controller, or other control logic. Inthe illustrated example, processor 1904 is connected to a bus 1902,although any communication medium can be used to facilitate interactionwith other components of computing component 1900 or to communicateexternally.

Computing component 1900 might also include one or more memorycomponents, simply referred to herein as main memory 1908. For example,preferably random access memory (RAM) or other dynamic memory, might beused for storing information and instructions to be executed byprocessor 1904. Main memory 1908 might also be used for storingtemporary variables or other intermediate information during executionof instructions to be executed by processor 1904. Computing component1900 might likewise include a read only memory (“ROM”) or other staticstorage device coupled to bus 1902 for storing static information andinstructions for processor 1904.

The computing component 1900 might also include one or more variousforms of information storage mechanism 1910, which might include, forexample, a media drive 1912 and a storage unit interface 1920. The mediadrive 1912 might include a drive or other mechanism to support fixed orremovable storage media 1914. For example, a hard disk drive, a floppydisk drive, a magnetic tape drive, an optical disk drive, a CD or DVDdrive (R or RW), or other removable or fixed media drive might beprovided. Accordingly, storage media 1914 might include, for example, ahard disk, a floppy disk, magnetic tape, cartridge, optical disk, a CDor DVD, or other fixed or removable medium that is read by, written toor accessed by media drive 1912. As these examples illustrate, thestorage media 1914 can include a computer usable storage medium havingstored therein computer software or data.

In alternative embodiments, information storage mechanism 1910 mightinclude other similar instrumentalities for allowing computer programsor other instructions or data to be loaded into computing component1900. Such instrumentalities might include, for example, a fixed orremovable storage unit 1922 and an interface 1920. Examples of suchstorage units 1922 and interfaces 1920 can include a program cartridgeand cartridge interface, a removable memory (for example, a flash memoryor other removable memory component) and memory slot, a PCMCIA slot andcard, and other fixed or removable storage units 1922 and interfaces1920 that allow software and data to be transferred from the storageunit 1922 to computing component 1900.

Computing component 1900 might also include a communications interface1924. Communications interface 1924 might be used to allow software anddata to be transferred between computing component 1900 and externaldevices. Examples of communications interface 1924 might include a modemor softmodem, a network interface (such as an Ethernet, networkinterface card, WiMedia, IEEE 802.XX or other interface), acommunications port (such as for example, a USB port, IR port, RS232port Bluetooth® interface, or other port), or other communicationsinterface. Software and data transferred via communications interface1924 might typically be carried on signals, which can be electronic,electromagnetic (which includes optical) or other signals capable ofbeing exchanged by a given communications interface 1924. These signalsmight be provided to communications interface 1924 via a channel 1928.This channel 1928 might carry signals and might be implemented using awired or wireless communication medium. Some examples of a channel mightinclude a phone line, a cellular link, an RF link, an optical link, anetwork interface, a local or wide area network, and other wired orwireless communications channels.

In this document, the terms “computer program medium” and “computerusable medium” are used to generally refer to media such as, forexample, memory 1908, storage unit 1920, media 1914, and channel 1928.These and other various forms of computer program media or computerusable media may be involved in carrying one or more sequences of one ormore instructions to a processing device for execution. Suchinstructions embodied on the medium, are generally referred to as“computer program code” or a “computer program product” (which may begrouped in the form of computer programs or other groupings). Whenexecuted, such instructions might enable the computing component 1900 toperform features or functions of the disclosed technology as discussedherein.

While various embodiments of the disclosed technology have beendescribed above, it should be understood that they have been presentedby way of example only, and not of limitation. Likewise, the variousdiagrams may depict an example architectural or other configuration forthe disclosed technology, which is done to aid in understanding thefeatures and functionality that can be included in the disclosedtechnology. The disclosed technology is not restricted to theillustrated example architectures or configurations, but the desiredfeatures can be implemented using a variety of alternative architecturesand configurations. Indeed, it will be apparent to one of skill in theart how alternative functional, logical or physical partitioning andconfigurations can be implemented to implement the desired features ofthe technology disclosed herein. Also, a multitude of differentconstituent component names other than those depicted herein can beapplied to the various partitions. Additionally, with regard to flowdiagrams, operational descriptions and method claims, the order in whichthe steps are presented herein shall not mandate that variousembodiments be implemented to perform the recited functionality in thesame order unless the context dictates otherwise.

Although the disclosed technology is described above in terms of variousexemplary embodiments and implementations, it should be understood thatthe various features, aspects and functionality described in one or moreof the individual embodiments are not limited in their applicability tothe particular embodiment with which they are described, but instead canbe applied, alone or in various combinations, to one or more of theother embodiments of the disclosed technology, whether or not suchembodiments are described and whether or not such features are presentedas being a part of a described embodiment. Thus, the breadth and scopeof the technology disclosed herein should not be limited by any of theabove-described exemplary embodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “component” does not imply that the components or functionalitydescribed or claimed as part of the component are all configured in acommon package. Indeed, any or all of the various components of acomponent, whether control logic or other components, can be combined ina single package or separately maintained and can further be distributedin multiple groupings or packages or across multiple locations.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives can be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

What is claimed is:
 1. An imaging system, comprising: a light source tooutput an optical signal modulated with a code; an image sensorcomprising a plurality of pixels; and a filter circuit to filter outsignals generated by one or more of the plurality of pixels from lightthat is not modulated with the code and to pass signals generated by oneor more of the plurality of pixels from light that is modulated with thecode.
 2. The imaging system of claim 1, wherein each of the plurality ofpixels comprise: a photodiode; the filter circuit; and a read outfield-effect transistor (FET).
 3. The imaging system of claim 1, whereinthe filter circuit is a digital filter circuit to filter one or moredigital signals output by the image sensor.
 4. The imaging system ofclaim 2, wherein the plurality of pixels comprises: a first plurality ofpixels, each of the first plurality of pixels to output a signalproportional to an intensity of light incident on a photodiode of thepixel; and a second plurality of pixels, each of the second plurality ofpixels to output a signal proportional to a change in intensity of lightincident on a photodiode of the pixel.
 5. The imaging system of claim 1,further comprising: a first filter circuit to output a signalproportional to an increase in intensity of light incident on one ormore of the plurality of pixels; and a second filter circuit to output asignal proportional to a decrease in intensity of light on one or moreof the plurality of pixels.
 6. The imaging system of claim 2, whereinthe plurality of pixels comprises: a first plurality of pixels, each ofthe first plurality of pixels to output a signal proportional to anintensity of light incident on a photodiode of the pixel; a secondplurality of pixels, each of the second plurality of pixels to output asignal proportional to an increase in intensity of light incident on aphotodiode of the pixel; and a third plurality of pixels, each of thethird plurality of pixels to output a signal proportional to a decreasein intensity of light incident on a photodiode of the pixel.
 7. Theimaging system of claim 1, wherein the imaging system is a structuredlight imaging system, and wherein the imaging system further comprises:a grating to shape the output optical signal modulated with a code. 8.The imaging system of claim 1, further comprising: a light source driverconfigured to output a modulated electrical signal to drive the lightsource based on the modulation code.
 9. The imaging system of claim 1,comprising: a first light source to output a first optical signalmodulated with a first code; and a second light source to output asecond optical signal modulated with a second code, a first filtercircuit to filter out signals generated from light that is not modulatedwith the first code; and a second filter circuit to filter out signalsgenerated from light that is not modulated with the second code.
 10. Theimaging system of claim 2, wherein the light source is to output aplurality of optical signals, each of the plurality of optical signalscorresponding to a respective waveband of light modulated with arespective code, and wherein each of the plurality of the pixels isconfigurable to detect one of the wavebands of light.
 11. The imagingsystem of claim 10, wherein each of the wavebands of light is phasemodulated, and wherein the filter circuit of each of the plurality ofthe pixels is configured to filter the wavebands of light based on theirphase.
 12. The imaging system of claim 3, comprising: a first lightsource to output a first color of light modulated with a first code; asecond light source to output a second color of light modulated with asecond code; a first digital filter circuit to filter out signalsgenerated from light that is not modulated with the first code; a seconddigital filter circuit to filter out signals generated from light thatis not modulated with the second code; and a digital filter circuit tocombine outputs of the first digital filter circuit and second digitalfilter circuit to generate a composite color image.
 13. The imagingsystem of claim 1, wherein the imaging system is a laser scan imagingsystem, wherein the light source comprises a laser line scanner, andwherein the imaging system comprises: a first filter circuit to output apeak signal when a phase of light incident on one or more of theplurality of pixels of the image sensor is a first value; a secondfilter circuit to output a peak signal when a phase of light incident onone or more of the plurality of pixels of the image sensor is a secondvalue; a third filter circuit to output a peak signal when a phase oflight incident on one or more of the plurality of pixels of the imagesensor is a third value; and a fourth filter circuit to output a peaksignal when a phase of light incident on one or more of the plurality ofpixels of the image sensor is a fourth value.
 14. The imaging system ofclaim 13, wherein a frequency of each of the plurality of pixels isoffset from a frequency of the optical signal.
 15. A method, comprising:generating a modulated optical signal at a light source of an imagingsystem, the modulated optical signal carrying a modulation code;configuring a filter circuit to filter out signals generated by one ormore pixels of a plurality of pixels of an image sensor from light thatis not modulated with the code and to pass signals generated by one ormore pixels of the image sensor from light that is modulated with thecode; receiving light at the image sensor; and using at least theconfigured filter circuit to filter the received light based on themodulation code.
 16. The method of claim 15, wherein each of theplurality of pixels comprises: a photodiode; the filter circuit; and aread out field-effect transistor (FET).
 17. The method of claim 15,wherein the filter circuit is a digital filter circuit to filter one ormore digital signals output by the image sensor.
 18. The method of claim16, further comprising: configuring a first set of the plurality ofpixels to output a signal proportional to an intensity of light incidenton a photodiode of the pixel; and configuring a second set of theplurality of pixels to output a signal proportional to a change inintensity of light incident on a photodiode of the pixel.
 19. The methodof claim 15, further comprising: configuring a first filter circuit tooutput a signal proportional to an increase in intensity of lightincident on one or more of the plurality of pixels; and configuring asecond filter circuit to output a signal proportional to a decrease inintensity of light on one or more of the plurality of pixels.
 20. Themethod of claim 16, further comprising: configuring a first set of theplurality of pixels to output a signal proportional to an intensity oflight incident on a photodiode of the pixel; configuring a second set ofthe plurality of pixels to output a signal proportional to an increasein intensity of light incident on a photodiode of the pixel; andconfiguring a third set of the plurality of pixels to output a signalproportional to a decrease in intensity of light incident on aphotodiode of the pixel.
 21. The method of claim 15, comprising:generating a first modulated optical signal at a first light source ofthe imaging system, the first modulated optical signal carrying a firstmodulation code; generating a second modulated optical signal at asecond light source of the imaging system, the second modulated opticalsignal carrying a second modulation code; configuring a first filtercircuit to filter out signals generated from light that is not modulatedwith the first code; and configuring a second filter circuit to filterout signals generated from light that is not modulated with the secondcode.
 22. The method of claim 15, wherein the imaging system is astructured light imaging system, and wherein the method furthercomprises: shaping the modulated optical signal by passing it through agrating of the imaging system.
 23. The method of claim 16, comprising:generating a first modulated optical signal at the light source, thefirst modulated optical signal having a first waveband and carrying afirst modulation code; generating a second modulated optical signal atthe first light, the second modulated optical signal having a secondwaveband and carrying a second modulation code; and configuring thefilter circuit of each of the plurality of pixels to detect lightcarrying the first modulation code or light carrying the secondmodulation code.
 24. The method of claim 23, wherein each of the opticalsignal is phase modulated, and wherein the filter circuit of each of theplurality of the pixels is configured to filter different wavebands ofincident light based on their phase.
 25. The method of claim 15, whereinthe imaging system is a laser scan imaging system, wherein the lightsource comprises a laser line scanner, and wherein the method furthercomprises: configuring a first filter circuit to output a peak signalwhen a phase of light incident on one or more of the plurality of pixelsof the image sensor is a first value; configuring a second filtercircuit to output a peak signal when a phase of light incident on one ormore of the plurality of pixels of the image sensor is a second value;configuring a third filter circuit to output a peak signal when a phaseof light incident on one or more of the plurality of pixels of the imagesensor is a third value; and configuring a fourth filter circuit tooutput a peak signal when a phase of light incident on one or more ofthe plurality of pixels of the image sensor is a fourth value.
 26. Animaging system, comprising: a light source to output an optical signal;an image sensor comprising a plurality of pixels; and one or more filtercircuits to add or subtract consecutively sampled signals to output acomposite signal including positive and negatives changes in anintensity of light incident on one or more of the plurality of pixels.27. The imaging system of claim 26, wherein the consecutive sampledsignals comprises a plurality of consecutively captured image frames,wherein the one or more filter circuits comprise one or more digitalfilter circuits to add or substract the plurality of consecutivelycaptured image frames to create a composite image frame, wherein thecomposite image frame displays positive and negative changes in theintensity of light incident on one or more of the plurality of thepixels during capture of the plurality of consecutively captured imageframes.
 28. The imaging system of claim 26, wherein each of theplurality of pixels comprise: a photodiode; the one or more filtercircuits; and a read out field-effect transistor (FET).